Most LLM prompts are mostly repetition. A support bot opens with the same 3,000-token instruction block, then appends a 50-token question. Across a fleet, every instance recomputes that block from scratch — unless the routing layer happens to keep sending it to the same machine.
On 2026-09-10, Amazon SageMaker Inference introduced prefix-aware routing, a strategy that reads the beginning of each request and consistently sends requests with the same beginning to the same instance. The AWS Machine Learning Blog post frames the problem plainly: prefix caching exists in vLLM and TensorRT-LLM, but random routing spreads a shared prefix so thinly across instances that no cache ever warms up.
The benchmark numbers, and what they depend on
AWS reports tests on Llama 3.1 70B Instruct across 7 ml.p5.48xlarge instances with vLLM prefix caching enabled, covering 16 configurations and both the native Invoke API and the OpenAI-compatible API. All runs completed at 100 percent success.
For 8,000-token shared prefixes sustained over an hour, P50 TTFT dropped 71–77 percent, P90 TTFT fell 33–37 percent, KV cache hit rate moved from roughly 25 percent to 82 percent, and throughput rose 15–16 percent. Short ShareGPT-style conversations showed smaller gains: P50 TTFT down 13–16 percent, throughput up 1.7–2.0 percent.
The pattern is the point. Longer shared prefixes mean more computation skipped per cache hit, so the win scales with how much of your prompt is fixed. Routing overhead measured 1.3–1.9 ms per request against model TTFT of 63–280 ms, and traffic stayed within 1 percent of an even split across instances.
Two safeguards worth understanding before you flip it on
Overload protection is the first. If a popular prefix lands on an instance already at your configured concurrency limit, the endpoint routes elsewhere and you eat a cache miss rather than a queue. That tradeoff is deliberate, and it means a single hot prefix cannot pin one machine.
The second is stability during scaling. When instances are added or removed, most requests keep going where they were; only a small fraction shifts. Caches are not invalidated on every scale event. If you have ever watched a fleet resize and wondered why latency spiked, this is the mechanism that addresses it.
Configuration is small, but the details bite
You set the strategy per production variant and can switch it by updating the endpoint configuration without redeploying the model. Two parameters matter: PrefixLength (1024–65536) and ConcurrencyThreshold (1–1024).
aws sagemaker create-endpoint-config \
--endpoint-config-name example-llm-config \
--production-variants '[{
"VariantName": "AllTraffic",
"ModelName": "example-llm-model",
"InitialInstanceCount": 3,
"InstanceType": "ml.p5.48xlarge",
"RoutingConfig": {
"RoutingStrategy": "PREFIX_AWARE",
"PrefixAwareRoutingConfig": {
"PrefixLength": 4096,
"ConcurrencyThreshold": 10
}
}
}]'Two practical traps stand out. For the native Invoke API, PrefixLength operates on raw bytes, so JSON whitespace and key ordering affect routing — inconsistent serialization can scatter requests that should share a cache. And sizing is a real decision: too short funnels unrelated traffic onto one instance and triggers overflow, too long lets trivial differences like a temperature value split requests apart. AWS suggests starting at your shared prefix length plus a modest buffer.
Multi-tenant setups get an escape hatch. Passing X-Amzn-SageMaker-Prefix-Aware-Id or prompt_cache_key combines with the prefix so identical text from different tenants lands on different instances.
What this means for your serving stack
Prefix-aware routing is a routing-layer change, not a container change. Your model container and serving framework stay as they are, and invocation APIs are unchanged. But the feature only pays off if prefix caching is actually enabled in your framework — vLLM has it on by default in recent versions, while others may need explicit configuration. You also need at least two instances for any of this to matter.
If you are already tuning inference economics, this belongs next to the work in our earlier piece on serving code models with a persistent kernel. Both are about the same underlying question: how much computation can you avoid repeating, and where does the system have to change to let you avoid it?
The honest limitation is that the gains are workload-shaped. RAG over a shared document, multi-turn chat, templated assistants, and code completion all share long prefixes and should benefit. A workload where every request is genuinely distinct will not. AWS recommends enabling detailed observability to track KV cache hit rates at the model level — that is the check that tells you whether this is working for your traffic rather than someone else’s benchmark.
Sources
AI-assisted summary compiled from the sources above, reviewed by a human before publishing.