Vercel AI Gateway: See Which Sessions Stayed on a Warm Cache

Summary: Vercel added a Sessions view to AI Gateway logs that groups requests into conversations and shows how session affinity routed each one, per the changelog published on October 18, 2026. Each row shows the client (Claude Code, Codex, Cursor), cost, models, providers, and how much of the input was cached across every turn. If you run multi-turn agents through the gateway, this turns an invisible cost lever into something you can actually inspect.

The problem it exposes

A provider’s prompt cache lives on the provider account that served the request. Per Vercel’s session affinity docs, when the next turn of a conversation lands on a different provider, that cache is cold and you pay full price for the prompt again. Until now, that failure mode was hard to see from the outside — you’d just notice the bill.

The new view groups requests by session and shows the cache hit rate, duration, and per-turn path, including which provider served each turn and which earlier ones. A session that stayed put reads more of each prompt from cache; one that bounced shows it immediately.

How affinity decides where a turn goes

Session affinity is on by default for text and chat requests. The gateway groups related requests, remembers which provider served the last one, and promotes that provider to the front of the list it already planned to try. Important boundaries from the docs:

It only reorders providers your routing filters allow. It never promotes a provider the gateway treats as down, and BYOK credentials stay in their own group.
If the pinned provider fails, normal fallback kicks in, and a request that fails after acceptance clears the pin.
Embeddings, images, and realtime requests aren’t covered — there’s no prompt cache to keep warm.

This fits the broader routing model I covered earlier in Vercel’s three-layer agentic infrastructure: the gateway keeps making provider decisions, and now you can audit one specific kind of decision.

Identifying your own sessions

By default, requests without an explicit ID get fingerprinted: Vercel hashes the opening system/developer message, the first non-system message, and the tool definitions with SHA-256 plus your team ID. That’s stable across turns and independent of tool order, but two conversations with identical openings will share a fingerprint.

For reliable grouping, send your own ID. Either option works:

const result = streamText({
  model: 'anthropic/claude-sonnet-5',
  prompt: 'Hello world',
  providerOptions: {
    gateway: { sessionId: 'sess_8f3KQ2vXr9mT' },
  },
});

Or set an x-session-id header on the OpenAI-compatible, Anthropic-compatible, or OpenResponses endpoints. Keep the value opaque (up to 256 characters), don’t include personal data, and don’t reuse one ID across unrelated work. Note that Claude Code sends its own header automatically, and a shared prompt_cache_key is treated as a shared-prefix label — it pins the provider, not a specific credential.

One more detail worth knowing: the gateway also forwards x-session-affinity upstream, so providers that use it for their own cache-cluster routing can act on it.

What I’d actually check in the view

The changelog says you can filter sessions by turns, client, model, provider, authentication, or cost, with single-turn sessions hidden until you clear a minimum-turns filter. A practical first pass: sort your agent traffic by cost, then look at whether the low-turn, high-cost sessions are the ones with poor cache hit rates — that’s usually a sign your client is mutating the prompt prefix mid-conversation, not a routing problem.

There’s a limit here too: the header doesn’t guarantee a cache hit, and Anthropic’s default cache lifetime is five minutes, so long-pausing agents will still pay the write premium again. If that’s your pattern, the docs suggest pairing affinity with a one-hour cache_ttl. The Sessions view won’t fix anything by itself — but it tells you which conversations are worth fixing.

Sources

Original post

View the original post on X