Summary: The article explains that provider tokenizers segment identical text differently, so list prices cannot determine real cost without provider-specific counts. TensorZero found Claude Opus 4.7 used 1.57 times OpenAI’s tokens for text and 2.65 times for tool definitions, while Firecrawl measured a 21.5x token reduction from raw HTML to markdown across 15 pages.
What Changed: There Is No Universal Token Count
You benchmarked two models, the lower price per million tokens won, and the invoice was larger anyway. Nothing was miscounted. The providers simply do not agree on how many tokens your text contains, and the model carrying the better sticker price was the one that manufactured more of them.
That disagreement has a name: LLM tokenization. It sits beneath every token bill you have ever paid. A token measures no length; it is the smallest unit a language model actually reads, prices, and charges against your context window. Every provider’s tokenizer holds a fixed vocabulary—typically 100,000 to 250,000 entries—built from a corpus by merging the most frequent character pairs repeatedly. A word that appears often stays whole as one token; a rare one is broken into fragments. Each provider trains on its own corpus with its own algorithm, so an identical string gets cut at different boundaries and the count differs.
That number is what you are billed on and what your context window is measured in. It is also the unit context engineering budgets around and the unit chunking cuts on. A chunk size calibrated on one model does not carry over to the next: set chunks at 512 tokens against GPT and the same strategy yields noticeably larger passages on Claude.
How Tokenization Works: Vocabulary Size and Content Type
Two factors explain nearly all of it: vocabulary size, and what the vocabulary was trained on. More entries means more whole words held and fewer pieces per word. Train a vocabulary mostly on English and English comes out efficient while everything else fragments. The Cambridge and Hong Kong team behind arXiv:2601.11518 follows both effects through ten production tokenizers.
Consider a single word. antidisestablishmentarianism costs 9 tokens on Claude, but only 5 on Gemini and Grok, and across models the whole spread runs from 4 to 9. Firecrawl’s own counts line up: for that word Gemini’s countTokens comes back with 5, while Anthropic’s comes back with 8.
A single word is a curiosity; the same effect across a million words is a budget line. On emoji and numeric content, the Cambridge team measured compression ratios diverging by 100% at the extremes, and by 20 to 50% on everything else. English essays, it also found, compress about 25% better than technical writing. What your tokens cost depends on the kind of text you send, not simply how much of it.
Any fair comparison of counts across models requires pushing identical bytes through each provider’s own counter. The cleanest public tokenizer comparison came from TensorZero in April 2026: identical inputs through each API, normalized against OpenAI. Those results, below, show how far token counts move with content type.
| Model | Text | YAML | JSON | Tool definitions |
|---|---|---|---|---|
gpt-5.4 |
1.00x | 1.00x | 1.00x | 1.00x |
gemini-3.1-pro-preview |
1.06x | 1.18x | 1.11x | 1.82x |
claude-sonnet-4-6 |
1.17x | 1.25x | 1.22x | 2.06x |
claude-opus-4-7 |
1.57x | 1.53x | 1.70x | 2.65x |
Read it in both directions. Travel across a row and each model deteriorates as the content grows more structured. Travel down a column and the distance between providers stretches from 1.57x on prose to 2.65x on tool schemas.
Practical Impact: What Effective Cost Means for Model Choice
Put the multiplier and the list price together and out comes the real cost per million tokens. Two facts defeat the naive comparison. First: on tool-heavy work claude-opus-4-7 costs 5.3x what gpt-5.4 does, though the list prices sit only 2x apart. Second: the ordering refuses to hold still. On text, Gemini comes in cheapest at $2.12 effective, then turns 46% more expensive than OpenAI the moment tool definitions go in.
No model is cheapest in the abstract. Cheapest holds only for a given content mix, and that is why “which model is cheaper” has no answer until you measure your own traffic. And if you have ever compared agents on token cost, those figures need exactly this correction.
Two further effects ride on top, both pushing the other way. Prompt caching gives repeated context a steep discount, while long-context tiers lift prices past a threshold. TensorZero’s numbers cover base input pricing alone, and the analysis here stays inside that boundary.
Structured Content and Raw HTML: Where Tokens Get Wasted
Structure is what works against you. Tokenizer vocabularies are built for prose, so a frequent word compresses into a single token. Structured text does the reverse: colons, indentation, quoted keys, braces, and closing tags all yield short tokens that carry almost no meaning. Take the JSON key "created_at": two quote marks and an underscore get paid for before the concept earns a single token.
Here is why so many agent developers keep running into this cost. Tool schemas are pure structure, and that explains both where they sit at the 2.65x end of the table and why a large pile of definitions is expensive before work even starts. Elsewhere the same effect shows up in miniature: at startup, merely advertising itself costs a Claude Code skill roughly 100 tokens, most of that overhead being frontmatter rather than instruction.
The most structured input anyone routinely feeds a model is raw HTML. Firecrawl built a benchmark around it: 15 real pages covering the kinds of things agents actually fetch, each one scraped in two versions—untouched rawHtml in one, markdown with onlyMainContent: true in the other—and then counted with each provider’s own tokenizer. GPT numbers come from running tiktoken locally with the o200k_base encoding, which is exact. Claude numbers come from Anthropic’s count_tokens endpoint minus the 7-token message wrapper. Gemini numbers come from Google’s countTokens on gemini-3.1-pro-preview, which adds no wrapper.
The totals: 4,103,706 GPT tokens as raw HTML against 190,817 as markdown, a 21.5x reduction. Median page: 19x, running from 2.6x on a Hacker News front page to 283x on Vercel’s homepage. All three tokenizers land in the same place: 21.5x on GPT, 20.8x on Claude, 22.5x on Gemini.
Cost is the obvious half of this story. Feasibility is the sharper half: five of the fifteen pages run past 200,000 tokens as raw HTML, which puts them entirely out of reach of a 200k context window. In markdown, not a single page came close. Format governs what you pay, and it also governs whether the request can be made.
Where do those enormous outliers come from? Both the Stripe reference and the Vercel homepage lean heavily on JavaScript, shipping inline scripts, serialized state, and styling inside the document itself. Barely any of that is content.
Then came the result that caught the researchers off guard. They expected cleaning a page to narrow the distance between tokenizers, on the theory that tag soup must be the hard case. What happened is that the two non-OpenAI tokenizers diverged. On raw HTML Claude took 1.168 tokens per GPT token; on markdown that rose to 1.210, a gap that widened on 13 of the 15 pages. Gemini moved the other way, from 1.249 down to 1.191. The reason: HTML is repetitive ASCII markup that any tokenizer handles about the same, so a cleaned page loses the easy material and keeps the prose—where vocabularies genuinely diverge. Which direction the gap moves depends on the vocabulary in play, and first principles will not tell you.
So no rule of thumb survives here. Format and tokenizer are two separate levers, and how the two behave together depends on the particular pair in front of you. The size of the format lever, though, does hold across all three tokenizers: roughly 21x, set against a tokenizer spread of about 1.2x. A better exchange rate is not something cleaning your input reliably buys, but an order of magnitude less to exchange is something it always buys.
What Non-English Content Costs: The Tokenizer Tax
Crawl anything beyond English and you meet the largest tokenization effect of all. At NeurIPS 2023, Petrov and colleagues showed that for equivalent content the least-served languages run more than twelve times the cost of English, and that the disparity takes shape during tokenization, in their words “well before the language model sees any data at all.” How much text fits in the window suffers the same premium, and so does latency.
Newer, larger vocabularies narrow the gap without closing it. A Ukrainian word needs 3.32 tokens under the older GPT-3.5 and GPT-4 vocabulary, and 1.98 on GPT-4o, where English takes 1.07. When your crawl targets non-English sites, the tokenizer you choose can be worth more than a few percentage points of list price.
One point to keep in mind before assuming the obvious: the Cambridge study turned up no significant correlation between how common a language is in Common Crawl and its compression ratio. The premium cannot be predicted from how widely a language is spoken. It has to be measured.
Measuring Your Own Content: The Code and the Traps
Stop guessing at the numbers. For an exact count, every provider offers either a count tokens API or a local library—and they contradict one another, which is the entire point.
# OpenAI: local and exact, no API call
import tiktoken
enc = tiktoken.get_encoding("o200k_base")
gpt_tokens = len(enc.encode(text))
# Anthropic: subtract the 7-token message wrapper for a bare-string count
import anthropic
client = anthropic.Anthropic()
claude_tokens = client.messages.count_tokens(
model="claude-sonnet-4-5",
messages=[{"role": "user", "content": text}],
).input_tokens - 7
# Google
from google import genai
gemini_tokens = genai.Client().models.count_tokens(
model="gemini-3.1-pro-preview", contents=text
).total_tokensTwo traps deserve names. First, Anthropic’s endpoint rejects payloads beyond roughly 100,000 characters, so long documents must be chunked and their counts summed; at 80,000-character chunks the boundary error measured 8 tokens in 24,452, or 0.03%. Second, skip the four-characters-per-token shortcut. The Cambridge team called that heuristic “inadequate and oversimplified”: randomly sampled English words measured 0.35 to 0.45 words per token, against the 0.75 the rule promises.
Once the counting is right, keep tracking it in production. Most LLM observability tools carry per-provider token spend as a standard metric, and nothing else will catch a shift in your content mix as it moves your effective price.
Takeaway: Fix the Format, Then Shop for a Model
A provider’s tokenizer is beyond your reach. Two things that matter more are not.
Choose the model by measured cost, not by list price. Take a representative slice of your real traffic, count it with all three counters, multiply each result by list price, then compare. When your workload leans on tools, the answer will likely differ from the pricing page’s.
Fix the format before you change anything else. Firecrawl’s benchmark prices that lever at 21.5x, against a tokenizer spread of only about 1.2x. For token efficiency, switching models is a rounding error beside not sending tag soup.
In practice, that comes down to three habits:
- Send markdown instead of HTML. Clean markdown is what Firecrawl’s
/scrapereturns by default, andonlyMainContent: truestrips navigation, footers, and inline scripts. There is the 21.5x. - Send excerpts rather than documents, when an answer is all you need. Instead of the whole page, the question and highlights formats give back grounded answers, or passages quoted verbatim.
- Stop retrieved pages from piling up. Whatever you scrape stays in context for every later turn, which is how a web research session inflates.
Provider-specific figures wearing the costume of a universal unit—that is what token counts are, and taking them at face value is how a cheaper model ends up costing you more. Before committing to a provider, run your own content through each counter. Then take a hard look at what you are sending. Across the fifteen pages tested, format counted for twenty times more than model choice.