BPBench: which LLMs best compress fresh human text?

A pre-trained LLM is a compressor: the better it models language, the fewer bits it needs to encode text it has never seen. BPBench measures exactly that — bits-per-byte (BPB) on fresh, provably human-written text published after each model's training cutoff. Nothing memorized, nothing gamed. Two steps: run a forward pass over the text, then compute bits-per-byte from the log probabilities. Lower = better compression = stronger language modeling (Huang et al. 2024: BPB tracks downstream capability near-linearly).

Trelis Research · July 2026 · @ronankmcgovern · code on GitHub

Exact BPB needs the model's probability for every token (teacher forcing), so the leaderboard covers the models that expose those probabilities: open weights (served with echo=true + logprobs) plus logprob-exposing APIs (Inkling via Tinker). Across the 13 models we could measure exactly, the best compressors are almost all from Chinese labs — DeepSeek, Kimi, GLM, Qwen3.7 — with NVIDIA's Nemotron the Western model inside the top cluster and Thinking Machines' Inkling just behind. Poolside's Laguna and Qwen's own newest open 27B trail their elders.

Lower = better compression. Compare within columns — domains differ in intrinsic predictability, so the macro averages the three slice ratios. 95% CIs bootstrapped over documents: the top eight (0.615–0.652) are one broad cluster. Nemotron measured at NVFP4 (its only serverless precision); Inkling scored on a 50-doc sample per slice via Tinker's logprobs; others at served default. Kimi K3 and Qwen's latest join once their weights release.

Mid- and post-training do change a model's probability distribution — they tend to make it more peaked and to increase BPB. Still, flagship-class models keep compressing better, and BPB appears to correlate reasonably well with capabilities.

Some new open models barely model raw human text

The strangest result: GPT-OSS needs ~1.5 bits/byte — 2.4× the frontier — and Gemma-4-31B reads ~2.4 (provisional; see note below). These models do fine on standard benchmarks, but on plain human prose their probability estimates fall apart. Perhaps in line with @jxmnop's analysis of GPT-OSS, they appear to lean heavily on synthetic/chat data relative to raw web corpus. The frontier cluster, Inkling, and Laguna all model raw text normally.

Estimating BPB when a model hides its logprobs

What about closed models — GPT-5.6 Sol, Claude Fable, Gemini Flash 3.6? Commercial APIs stopped exposing prompt logprobs after Nicolas Carlini showed logprobs let you recover parts of the model (the final embedding layer). So exact BPB is off the table; the only route is estimating probabilities by sampling one token at a time — repeatedly, at many positions. We validated three ways of doing that on open models, where the true logprobs let us score each estimator. TL;DR: we did not find a robust way to estimate commercial-model logprobs — but the ideas are shared here because someone can likely improve on them.

1. The completions estimate

Re-sample the next token at each position (32 times) and use the match rate to estimate the probability. "Completions" = decoding on the raw string, no chat template. This works surprisingly well: DeepSeek, Kimi, GLM and other frontier models land close to their true BPB. Weaker models under-read, because of a hard resolution limit: with n=32 samples, a word that never appears can only be scored as "roughly 1/64" — a 6-bit floor. Positions where the true word costs 8, 12, 16 bits all get recorded as ~6. In our per-position data, positions whose true cost averages 16 bits are recorded at 5.98. Weak models are weak precisely where this estimator cannot see — which is why Laguna's completions estimate reads 0.11–0.17 below its true BPB, and why more samples barely help (the floor rises one bit per doubling of n). Unfortunately, OpenAI and others have been chat-only since roughly GPT-4 (the last completions model was gpt-3.5-turbo-instruct), so for commercial models a chat template is unavoidable.

2. The plain-chat estimate

Put the text in the user turn and ask for the next word, 32 times per position. This does not work well: the first token of an assistant reply always carries probability mass for starting a summary, a rebuttal, a rephrasing — it is not a true continuation. Measured token-exactly (on Inkling, whose Tinker API exposes logprobs), the chat frame costs about +2.7 bits/word at the response start. Worse, the resulting bias differs by model (−0.01 to −0.19 bits/byte across our set — an RLHF fingerprint, not a measurement), so it cannot be calibrated away.

3. The echo-chat estimate

To mitigate the boundary problem: provide the text in the user turn and append Begin exactly with: "<the last 8 words of the text>", then extract the word the model produces right after the echo. The prompt wording matters enormously: asking the model to "repeat the last 8 words" without quoting them gets ~2% compliance; quoting the exact string gets 92–100%. The echo walks the model back into the document's flow — token-exact logprobs on Inkling confirm the conditional returns to normal (median 2.34 bits/word vs 2.31 raw, from 4.74 cold). The estimator's bias collapses to something near-uniform across models, but a bias remains, likely at least because:

• the echo format is not a pure completion, and may impart bias;
• commercial APIs run hidden samplers (top-k, top-p) that truncate and shape the distribution in ways that are hard to determine from outside;
• the 6-bit floor applies here too — it is still a sampled estimate.

Whether plain-chat or echo-chat, it is crucial to turn off reasoning AND instruct the model not to think: thinking tokens sit between the user turn and the visible reply and distort continuation estimates. We log the reasoning rate per model via API telemetry, and separately measure echo adherence (did the reply start with the quoted words). If either is poor, the estimate can't be trusted — low-adherence models are greyed and excluded from the fit below.

It would have been very nice to overlay estimates for Fable, Gemini Flash 3.6 and GPT-5.6 Sol here — but with current methods it would not be a reliable comparison. Until someone improves on these estimators, BPB for open-weight models stands as the indicator of model strength.

Read tiers, not rankings

Even exact BPB is loose: it scores the deployed checkpoint, and post-training shifts it upward (on our Hansard slice, Llama-3-8B moves from 0.624 as a base model to 0.712 as an instruct model — a single-pair, single-slice measurement, but a gap larger than the whole frontier spread). So read the leaderboard as a broad frontier cluster plus clearly separate tiers, never as a point ranking of pretraining quality.

Where this came from

The inspiration is Ryan Greenblatt's no-CoT math time horizon — measuring what models can do in a single forward pass, no chain-of-thought. The challenge there is benchmarking task difficulty reliably. We went back to information theory instead: don't grade tasks, ask how novel unseen human text appears to the model. Difficulty calibration is replaced by a ground truth every model faces identically — the actual next words.

The data problem: fresh AND human

This requires text that is human-written and post-dates every model's training cutoff — and in 2026 roughly half of new web text is AI-written, so scoring models on it would measure self-similarity, not quality. We use provenance, not detectors (detectors select for text models find surprising — biasing the exact metric being measured):

• Transcribed parliamentary speech (US Congressional Record, UK Hansard floor debate) — words spoken aloud by humans, with written insertions filtered out. Imperfect: some speeches may be drafted with AI assistance, but the spoken channel is the strongest cheap provenance signal available.
• A private held-out prose slice to catch future benchmark gaming: a model whose gap between the public and private prose widens across releases is the flag. It ranks models consistently with the public slices today.

Public slices on Hugging Face: Congressional Record · Hansard. The private slice is held out.

Why we don't report code (yet)

We'd like a fresh-code axis, but sourcing genuinely fresh, provably human-written code needs careful review of provenance, and this round was time-constrained — we didn't get to find and vet good fresh-code sources properly. Rather than ship a number we can't stand behind, code waits for a future round.

Method notes

• Exact BPB: teacher-forced logprobs over every token of every document (Hansard 147 + Congress 259 + private 51 docs), byte-counted on the raw UTF-8 text. Bootstrap 95% CIs over documents.
• Estimators: uniform design — 3 evenly-spaced positions per document across all 457 docs (1,365 positions), n=32 samples per position, temperature 1, reasoning disabled and verified per model via API telemetry (reasoning-token counts). A position only counts if ≥16 of its 32 samples adhere (keeps the sampling floor comparable); adherence and coverage are reported per model.
• Gemma-4 footnote: its ~2.4 figure is a single-scorer (vLLM) measurement. The identical pipeline scores Gemma-2/3 normally, arguing against a tokenization (BOS) artifact, but vLLM's new Gemma-4 support could not be independently cross-checked — treat as provisional.
• Instruct checkpoints, one provider per measurement; serving precision not always disclosed — read results as "deployed checkpoint quality."