Summary: Ai2 compared the 7B-scale Olmo 3 and Olmo Hybrid models. Both used the same data, tokenizer, and training recipe. Token-level results favored hybrids on content words and semantic-state tasks, but the advantage faded on long verbatim copies and closing brackets.
Where exactly do hybrid language models outperform pure transformers? On June 25, 2026, Ai2 took up that hot question with unusually sharp instruments: a blog post titled “Which tokens does a hybrid model predict better?”, released together with a technical report, “Comparing Transformers and Hybrid Models at the Token Level” — written by Yanhong Li and William Merrill and posted to arXiv on June 18.
Put a hybrid next to a pure transformer and compare average losses, and all you get is one aggregate figure. This work drills down to individual tokens instead. Both models are given the same passage to read; at every position, the researchers record how much probability each model grants the correct next token. Subtracting the two yields a per-token loss gap — a positive value favors the hybrid, a negative one favors the transformer — and sorting the tokens by type then exposes the patterns.
Study Design: One Variable — the Architecture
At 7B scale, the study pits Olmo 3, a pure transformer, against Olmo Hybrid. Both were deliberately trained on the same data, with the same tokenizer and the same recipe, so any difference in how they predict must come from the architecture and nothing else. Alongside that pair, the team built three 1B models: a pure transformer, a hybrid, and a purely recurrent network carrying no attention whatsoever. The 7B pair establishes that an architecture gap exists; the 1B trio makes attribution possible — a pattern that persists in the attention-free model belongs to the recurrent layers, while one that vanishes there was attention’s doing.
The material covers prose and structured text alike — news articles, Wikipedia, books, and papers on the prose side; Python, HTML, and LaTeX on the structured side. Average gaps were computed after sorting tokens into groups by part of speech and structural role, and a regression that adjusts for confounds like token rarity and repetition verified that the patterns stand apart from statistical noise.
Content Words Are Where the Hybrid Pulls Ahead
Across most token types the hybrid posts the lower loss, and the margin is widest on content words: for nouns, verbs, and adjectives the loss gap runs around 0.04, while function words sit near 0.02. Gaps on adverbs, adjectives, and the existential “there” stand out as especially large — the hybrid’s strength tracks semantic understanding, not surface form. Articles, prepositions, and the other connective tissue of a sentence can largely be guessed from local context, and on those tokens the hybrid’s lead drops by half.
The paper’s synthetic probes push the picture further: on pronoun-memory and entity-tracking tasks, hybrids come out ahead. In the authors’ reading, recurrent layers excel at holding a semantic state that carries across the whole document, which is why they matter most for tokens whose prediction hinges on what the document means — frequently the exact tokens that determine what a sentence says.
Where the Edge Fades: Verbatim Copies and Closing Brackets
Two token kinds strip the hybrid of nearly all its advantage. One is verbatim copying: when an n-gram reappears word-for-word from earlier in the passage, the longer the repeated run, the closer the gap slides toward zero. The other is closing delimiters: on opening brackets the hybrid still leads, yet on closing brackets that lead evaporates — and the same holds in natural language, code, and markup alike, indicating bracket matching is something attention alone already handles.
The 1B results draw the line even sharper. Given non-repeating, meaning-bearing tokens, the hybrid and the pure recurrent model each defeat the transformer, the hybrid on top. Handed repeated tokens, the pure recurrent model trails both. Everything points the same direction: attention is where copying skill comes from; tracking semantic state is what recurrent layers do best.
What This Means for Building Models
Think of average loss as a telescope: it reports the total and hides the breakdown. What this study hands developers is a microscope — token losses, sorted by category, are available early in pretraining, well before a full training run has finished. Any team weighing how many attention layers against how many recurrent layers can now treat “hybrids are stronger” as a number that can be measured and tracked rather than a slogan. Bets on architecture at frontier scale burn months of compute; a diagnostic that surfaces fine-grained ability differences mid-run changes the economics of those bets, since a bad mix gets spotted before the bill comes fully due.
The results also map a division of labor inside the model itself. Carrying the document’s semantic state falls to the recurrent layers, which earn their keep on meaning-bearing tokens; copying earlier text and matching brackets are the mechanical jobs attention takes care of. A balanced hybrid is not simply “more model” — it stacks two distinct competencies, and there is finally a way to check that a given training run is actually acquiring both.
For the open-models community there is a further point: everything runs on Olmo weights and a training recipe that Ai2 released openly, so the analysis is there for anyone to reproduce or extend. The controlled comparison works only because data and recipe can be aligned completely — an option closed models simply do not offer.