On June 25, 2026, Ai2 published a blog post, “Which tokens does a hybrid model predict better?”, alongside a technical report uploaded to arXiv on June 18, “Comparing Transformers and Hybrid Models at the Token Level” by Yanhong Li and William Merrill. Together they tackle a hot question with an unusually sharp instrument: where, exactly, do hybrid language models beat pure transformers?
Comparing the average loss of a hybrid against a pure transformer yields one blunt number. This study compares token by token instead: both models read the same passage, and the researchers measure the probability each assigns to the true next token. The loss gap is computed per token — positive means the hybrid predicted better, negative means the transformer did — and tokens are then grouped by type to reveal patterns.
Study Design: A Comparison That Differs Only in Architecture
The study uses Olmo 3, a 7B pure transformer, and Olmo Hybrid. The two deliberately share training data, tokenizer, and training recipe, so prediction differences can be attributed cleanly to architecture alone. The team also prepared three 1B models — a pure transformer, a hybrid, and a pure recurrent model with no attention at all — to isolate what each component contributes. Where the 7B pair shows that an architecture difference exists at all, the 1B trio is what lets the researchers assign credit: if a pattern survives in the model with no attention, it belongs to the recurrent layers; if it disappears, attention was doing the work.
The corpus spans prose and structured text: prose from news articles, Wikipedia, books, and papers; structured text from Python, HTML, and LaTeX. Tokens were grouped by part of speech and structure to compute average gaps, and a regression controlling for confounds such as token rarity and repetition confirmed the patterns are not statistical artifacts.
The Hybrid Wins on Content Words
The hybrid achieves lower loss on most token types, and its largest advantage shows up on content words: nouns, verbs, and adjectives show a loss gap of roughly 0.04, against about 0.02 for function words. Adverbs, adjectives, and the existential “there” display especially large gaps, tying the hybrid’s strength to semantic understanding rather than surface form. Function words — articles, prepositions, the connective tissue of a sentence — are comparatively predictable from local context, and there the hybrid’s lead shrinks by half.
Synthetic probe experiments in the paper go further: hybrids are favored on pronoun-memory and entity-tracking tasks. The authors’ interpretation is that recurrent layers are good at maintaining a semantic state that persists across the document, so they help most on tokens whose prediction depends on what the document means — often the very tokens that decide the meaning of a sentence.
Where It Loses: Verbatim Copying and Closing Brackets
The hybrid’s edge nearly vanishes on two kinds of tokens. The first is verbatim copying: for n-grams repeated word-for-word from earlier in the passage, longer repeated runs push the gap toward zero. The second is closing delimiters: the hybrid still wins on opening brackets, but its advantage disappears on closing ones, and the pattern holds across natural language, code, and markup — plain attention is already enough for bracket matching.
The 1B experiments cut the conclusion even cleaner. On non-repeating, meaning-bearing tokens, both the hybrid and the pure recurrent model beat the transformer, with the hybrid best. On repeated tokens, the pure recurrent model falls behind both. The comparison points one way: copying ability comes mostly from attention, while semantic state tracking is the recurrent layer’s home turf.
Why This Matters for Model Development
Average loss is a telescope: it shows you the total score, not the structure. This study hands developers a microscope — categorized token losses can be computed early in pretraining, long before a full training run finishes. For teams deciding how to ratio attention layers to recurrent layers, the method turns “hybrids are stronger” from a slogan into a measurable, trackable signal. Architecture bets at frontier scale cost months of compute; a diagnostic that reads out fine-grained ability differences mid-run changes the economics of those bets, because a wrong mix can be caught before it is fully paid for.
The findings also sketch a division of labor inside the model. Recurrent layers carry the semantic state of the document and pay off on the tokens that carry meaning; attention handles the mechanical jobs of copying earlier text and matching brackets. A well-balanced hybrid is not just “more model” — it is two different competencies stacked, and now there is a way to check whether a given training run is actually getting both.
There is an extra layer for the open-models community: the experiment runs entirely on Ai2’s openly released Olmo weights and training recipe, so anyone can reproduce and extend the analysis. The controlled comparison is only possible because data and recipe can be fully aligned — something you simply cannot do on closed models.
Sources
- Which tokens does a hybrid model predict better? — Ai2
- Comparing Transformers and Hybrid Models at the Token Level — arXiv
AI-assisted summary compiled from the sources above, reviewed by a human before publishing.