Summary: Prompt caching is now table stakes across the major LLM APIs, but the pricing and hit rates vary wildly depending on how you structure requests. Firecrawl’s October 18, 2026 write-up by Hiba Fathima pulls together provider docs, gateway traffic data, and break-even math into one practical reference. The core lesson: caching is a byte-exact prefix match, and most misses are self-inflicted configuration problems.
Same model, 3x difference in hit rate
The most striking data point comes from the AI gateway Requesty, which measured hit rates across its April 2026 traffic. Requests to Claude through Anthropic’s own API read 77% of input tokens from cache. The same Claude models through Google’s Vertex read 24%. The gateway attributed the gap to configuration, not model behavior.
That gap is why caching deserves the same operational attention as latency or error rate. Anthropic’s Claude Code team alerts on cache hit rate and treats a drop as an incident. OpenAI shipped a miss-diagnostics field and a dashboard in September 2026. If you run agents in production and don’t know your hit rate, you’re flying blind on a large chunk of your bill. I’ve covered the Bedrock angle of this before in where Bedrock’s 90% input savings actually come from.
Why only prefixes work
Prompt caching stores KV tensors — the key and value vectors every layer computes for each input token — after the request ends, then reuses them when a later request starts with the same bytes. The lookup is hash-based and strictly prefix-matched because attention is causal: token 500’s vectors depend on tokens 1 through 499. Change token 12 and everything from there onward is invalidated. Change only the last message and the whole prefix stays valid.
So the design rule is simple: immutable content first, volatile content last. What breaks it in practice is mundane — timestamps at the top of a prompt, tool definition tweaks, switching models or effort settings mid-conversation, unstable JSON serialization, or idle gaps longer than the cache lifetime.
The pricing math
Read prices are steeply discounted: 0.1x base input on Anthropic’s standard models (0.05x on Opus 5.5 and Sonnet 5.5, 0.025x on Fable 5.1), 0.1x on GPT-5.6 and later (0.05x on GPT-6.1 Sol), and discounted-but-unguaranteed on Gemini. Writes cost a premium: 1.25x for Anthropic’s 5-minute tier or 2x for the 1-hour tier, 1.25x on OpenAI, and nothing extra on Gemini’s implicit caching.
That write premium sets the break-even: one reuse pays for the 1.25x tier, two for the 1-hour option. Since agentic coding tasks consumed roughly 1,000x more tokens than code chat in Longju Bai and colleagues’ April 2026 SWE-bench study — with input tokens driving most of it — multi-turn agents clear that bar easily. LangChain’s June 2026 agent evals measured 49–80% lower token cost across Anthropic, OpenAI, and Gemini once caching was in play.
The provider quirks that bite
Each API has details worth knowing before you ship:
- Anthropic counts the 5-minute lifetime from the start of the request, not the end. A 4-minute streamed response leaves about a minute for the follow-up to hit. And parallel requests with the same prefix all pay the write — send one, wait for it to start responding, then fan out.
- OpenAI caches state on individual machines; above 15 requests per minute traffic can overflow to machines without the entry. A stable
prompt_cache_keyhelps routing on pre-5.6 models. Mid-conversation effort changes need aconfiguration_updateitem rather than flipping the request-level setting. - Gemini caches implicitly with a 4,096-token minimum on current models, so short prompts never cache and no hit is guaranteed.
The cheapest optimization left in most stacks is measuring hit rate per request and alerting when it drops. Everything else here follows from that number.