LLM Tokenization: Why the Same Text Costs 2.65x More on Different Models

What Changed: Token Counts Are Not Universal

You benchmarked two models, picked the one with the lower price per million tokens, and the invoice came in higher anyway. Nothing was miscounted. The two providers simply disagree about how many tokens your text contains, and the one with the better sticker price turned your content into more of them.

That disagreement is LLM tokenization, and it is the layer under every token bill you have ever paid. A token is not a unit of length; it is the smallest unit a language model actually reads, prices, and counts against your context window. Each provider’s tokenizer holds a fixed vocabulary—typically 100,000 to 250,000 entries—learned from a training corpus by repeatedly merging the most frequent character pairs. Common words survive as single tokens; rare ones get chopped into fragments. Because each provider trains on its own corpus with its own algorithm, the same string lands on different boundaries, and you get a different count.

That count is the billing unit and the context-window unit. It is also the unit context engineering budgets in, and the unit chunking splits on. A chunk size tuned on one model does not transfer to another: if you sized your chunks at 512 tokens against GPT, the same chunking strategy produces meaningfully larger passages on Claude.

How Tokenization Works: Vocabulary and Content Type

Vocabulary size and training corpus drive almost all of it. A larger vocabulary holds more whole words, so it needs fewer pieces per word. A vocabulary trained heavily on English handles English efficiently and fragments everything else. The Cambridge and Hong Kong team behind arXiv:2601.11518 traces both effects across ten production tokenizers.

Take one word: antidisestablishmentarianism becomes 9 tokens on Claude and 5 on Gemini and Grok, with the full range running from 4 to 9 depending on the model. Firecrawl’s own counts agree: Gemini’s countTokens returns 5 for that word and Anthropic’s returns 8.

One word is a curiosity. The same effect across a million words is a budget line. The same study found compression ratios differing by 100% between the extremes on emoji and numeric content, and by 20 to 50% on everything else. It also found that English essays compress about 25% better than technical writing. Your token cost depends on what kind of text you send, not only how much.

To compare token counts across models fairly, you have to send the exact same bytes through each provider’s own counter. TensorZero ran the cleanest public tokenizer comparison in April 2026, feeding identical inputs through each API and normalizing against OpenAI. The results, shown below, reveal how much token counts vary by content type.

Model	Text	YAML	JSON	Tool definitions
gpt-5.4	1.00x	1.00x	1.00x	1.00x
gemini-3.1-pro-preview	1.06x	1.18x	1.11x	1.82x
claude-sonnet-4-6	1.17x	1.25x	1.22x	2.06x
claude-opus-4-7	1.57x	1.53x	1.70x	2.65x

Read the rows and the columns. Across the rows, every model gets worse as content gets more structured. Down the columns, the spread between providers widens from 1.57x on prose to 2.65x on tool schemas.

Practical Impact: Effective Cost and Model Choice

Multiply the multiplier by the list price and the real cost per million tokens falls out. Two things break the naive comparison. First, claude-opus-4-7 ends up costing 5.3x what gpt-5.4 costs on tool-heavy work even though their list prices are only 2x apart. Second, the ranking is not stable: Gemini is the cheapest option on text at $2.12 effective, then becomes 46% more expensive than OpenAI once you send tool definitions.

There is no single cheapest model. There is only a cheapest model for a given content mix, which is why “which model is cheaper” is unanswerable without measuring your own traffic. If you have compared agents on token cost, this is the correction to apply to those numbers.

Two more effects sit on top and both cut the other way. Prompt caching discounts repeated context heavily, and long-context tiers raise prices past a threshold. TensorZero’s figures cover base input pricing only, and so does the reasoning here.

Structured Content and Raw HTML: The Token Killers

Structure is the enemy. Prose is what tokenizer vocabularies are built for, so common words compress into single tokens. Structured text is the opposite: braces, quoted keys, colons, indentation, and closing tags produce short tokens carrying almost no meaning. A JSON key like "created_at" spends tokens on two quote marks and an underscore before it spends any on the concept.

This is the mechanism behind a cost most agent developers have already met. Tool schemas are pure structure, which is why they land at the 2.65x end of the table and why loading many of them is expensive before an agent does any work. The same effect shows up in miniature elsewhere: a Claude Code skill costs roughly 100 tokens just to advertise itself at startup, and that overhead is mostly frontmatter rather than instruction.

Raw HTML is the most structured input anyone routinely hands a model. Firecrawl ran a benchmark: they took 15 real pages spanning the kinds of things agents actually fetch, scraped each one twice—once as untouched rawHtml and once as markdown with onlyMainContent: true—then counted both with each provider’s own tokenizer. GPT counts come from tiktoken with the o200k_base encoding, which runs locally and is exact. Claude counts come from Anthropic’s count_tokens endpoint with the 7-token message wrapper subtracted. Gemini counts come from Google’s countTokens on gemini-3.1-pro-preview, which adds no wrapper.

The aggregate: 4,103,706 GPT tokens as raw HTML against 190,817 as markdown, a 21.5x reduction. Median per page was 19x, and the spread ran from 2.6x on a Hacker News front page to 283x on Vercel’s homepage. The reduction holds on every tokenizer: 21.5x on GPT, 20.8x on Claude, 22.5x on Gemini.

The cost story is the obvious one. The feasibility story is the sharper one: five of the fifteen pages exceed 200,000 tokens as raw HTML, which means they do not fit in a 200k context window at all. As markdown, not one page came close. Format is not only what you pay, it is whether the request is possible.

Why the enormous outliers? The Stripe reference and the Vercel homepage are both JavaScript-heavy pages that ship inline scripts, serialized state, and styling in the document. Almost none of that is content.

Now the part that surprised the researchers. They expected cleaning the page to shrink the gap between tokenizers, since tag soup should be the hard case. Instead the two non-OpenAI tokenizers moved in opposite directions. Claude went from 1.168 tokens per GPT token on raw HTML to 1.210 on markdown, a gap that widened on 13 of the 15 pages. Gemini went the other way, from 1.249 down to 1.191. The explanation is that HTML is repetitive ASCII markup that every tokenizer handles roughly equally, so cleaning a page strips out the easy part and leaves the prose, where vocabularies genuinely differ. Which way that lands depends on the specific vocabulary, and it is not predictable from first principles.

So there is no rule of thumb here. Format and tokenizer are separate levers, and how they interact depends on the pair you happen to be using. What does hold across all three tokenizers is the size of the format lever: roughly 21x, against a tokenizer spread of about 1.2x. Cleaning your input does not reliably buy a better exchange rate, but it always buys you an order of magnitude less to exchange.

Non-English Content: The Tokenizer Tax

If you crawl beyond English, this is the largest tokenization effect you will meet. Petrov and colleagues showed at NeurIPS 2023 that the worst-served languages cost more than twelve times what English costs for equivalent content, and that the disparity is created at the tokenization stage, in their words “well before the language model sees any data at all.” The same premium hits latency and how much text fits in the window.

Newer tokenizers with larger vocabularies narrow the gap without closing it. GPT-4o needs 1.98 tokens per Ukrainian word against 1.07 for English, while the older GPT-3.5 and GPT-4 vocabulary needs 3.32. If your crawl targets non-English sites, tokenizer choice is worth more than a few percentage points of list price.

One caveat against assuming the obvious: the Cambridge study found no significant correlation between a language’s compression ratio and how common it is in Common Crawl. You cannot predict the premium from how widely spoken a language is. You have to measure it.

Measuring Your Own Content: Code and Caveats

Stop estimating. Every provider ships a count tokens API or a local library that is exact, and they disagree with each other, which is the whole point.

# OpenAI: local and exact, no API call
import tiktoken
enc = tiktoken.get_encoding("o200k_base")
gpt_tokens = len(enc.encode(text))

# Anthropic: subtract the 7-token message wrapper for a bare-string count
import anthropic
client = anthropic.Anthropic()
claude_tokens = client.messages.count_tokens(
    model="claude-sonnet-4-5",
    messages=[{"role": "user", "content": text}],
).input_tokens - 7

# Google
from google import genai
gemini_tokens = genai.Client().models.count_tokens(
    model="gemini-3.1-pro-preview", contents=text
).total_tokens

Two traps worth naming. Anthropic’s endpoint rejects payloads past roughly 100,000 characters, so long documents need chunking and summing; at 80,000-character chunks the boundary error was measured at 8 tokens in 24,452, or 0.03%. And do not reach for the four-characters-per-token shortcut. The Cambridge team called that heuristic “inadequate and oversimplified”, measuring 0.35 to 0.45 words per token on randomly sampled English words against the 0.75 the rule promises.

Once you are counting properly, track it in production. Per-provider token spend is a standard metric in most LLM observability tools, and it is the only way to catch a content-mix shift moving your effective price.

Takeaway: Fix Format First, Then Choose Model

You cannot change a provider’s tokenizer. You can change two things that matter more.

Pick the model against measured cost, not list price. Run a representative sample of your real traffic through all three counters, multiply by list price, and compare that. If your workload is tool-heavy, the answer will likely differ from the one the pricing page implies.

Fix the input format first. The Firecrawl benchmark puts this at 21.5x, against a tokenizer spread of roughly 1.2x. For token efficiency, switching models is a rounding error next to not sending tag soup.

In practice that means three things:

• Send markdown, not HTML. Firecrawl’s /scrape returns clean markdown by default, with onlyMainContent: true stripping navigation, footers, and inline scripts. That is the 21.5x.
• Send excerpts, not documents, when you only need an answer. Question and highlights formats return grounded answers or verbatim passages instead of the whole page.
• Keep retrieved pages from accumulating. Scraped content stays in context for every subsequent turn, which is exactly how web research inflates a session.

Token counts are provider-specific measurements dressed up as a universal unit, and treating them as universal is how a cheaper model produces a larger bill. Measure your own content through each provider’s counter before you commit to one. Then look at what you are sending. On the fifteen pages tested, the format of the input mattered twenty times more than the choice of model.

Sources

• Original source

AI-assisted summary compiled from the sources above, reviewed by a human before publishing.