Summary: Amazon SageMaker Inference added prefix-aware routing on September 10, 2026 so requests with the same configured prompt prefix reach one instance and reuse KV cache. AWS reports large latency and cache-hit gains for long shared prefixes on Llama 3.1 70B, but distinct workloads see little benefit and vLLM prefix caching must actually be enabled.
Most LLM prompts are mostly repetition. The same 3,000-token instruction block leads a support bot’s request, and only then comes a 50-token question. Looked at across a fleet, every instance recalculates that block on its own — unless the routing layer happens to keep delivering it to one machine.
Amazon SageMaker Inference brought out prefix-aware routing on 2026-09-10. The strategy inspects the start of each request and reliably sends requests that share that start to a single instance. The AWS Machine Learning Blog post states the problem without dressing it up: prefix caching already exists in vLLM and TensorRT-LLM, yet random routing thins a shared prefix across instances so much that no cache ever warms up.
What the benchmark numbers show, and what they hinge on
AWS describes tests on Llama 3.1 70B Instruct spanning 7 ml.p5.48xlarge instances with vLLM prefix caching turned on, covering 16 configurations and exercised through both the native Invoke API and the OpenAI-compatible API. Every run finished at 100 percent success.
For 8,000-token shared prefixes held over an hour, P50 TTFT fell 71–77 percent and P90 TTFT 33–37 percent, KV cache hit rate climbed from roughly 25 percent to 82 percent, and throughput gained 15–16 percent. Short ShareGPT-style conversations moved less: P50 TTFT down 13–16 percent, throughput up 1.7–2.0 percent.
The shape of that result is the whole point. The longer the shared prefix, the more computation each cache hit lets you skip, so the benefit grows with the fixed share of your prompt. Routing overhead came in at 1.3–1.9 ms per request against model TTFT of 63–280 ms, and traffic stayed within 1 percent of an even split across instances.
Two safeguards to understand before you flip it on
Overload protection comes first. When a popular prefix reaches an instance that has already hit your configured concurrency limit, the endpoint sends it elsewhere, so you take a cache miss instead of a queue. That trade is deliberate, and it keeps one hot prefix from pinning a single machine.
The second is stability while scaling. As instances are added or removed, most requests keep going to the machine they were already using, and only a small fraction moves. Caches survive each scale event rather than being wiped. If you have ever watched a fleet resize and wondered why latency spiked, this is the mechanism aimed at that.
A small configuration surface, with sharp edges
You set the strategy per production variant, and switching it is a matter of updating the endpoint configuration — no model redeployment. Two parameters matter: PrefixLength (1024–65536) and ConcurrencyThreshold (1–1024).
aws sagemaker create-endpoint-config \
--endpoint-config-name example-llm-config \
--production-variants '[{
"VariantName": "AllTraffic",
"ModelName": "example-llm-model",
"InitialInstanceCount": 3,
"InstanceType": "ml.p5.48xlarge",
"RoutingConfig": {
"RoutingStrategy": "PREFIX_AWARE",
"PrefixAwareRoutingConfig": {
"PrefixLength": 4096,
"ConcurrencyThreshold": 10
}
}
}]'Two practical traps stand out. Under the native Invoke API, PrefixLength reads raw bytes, so JSON whitespace and key ordering feed into routing decisions — serialize the same prompt two ways and the requests can land on different instances. Sizing is a real decision as well: set it too short and unrelated traffic funnels onto one instance and triggers overflow; set it too long and trivial differences such as a temperature value split apart requests that belonged together. AWS suggests starting at your shared prefix length plus a modest buffer.
Multi-tenant setups get an escape hatch. Passing X-Amzn-SageMaker-Prefix-Aware-Id or prompt_cache_key combines with the prefix, so identical text coming from different tenants lands on different instances.
What it means for the rest of your serving stack
Prefix-aware routing changes the routing layer, not the container. Your model container and serving framework stay as they are, and the invocation APIs are unchanged. The feature only pays off, though, if prefix caching is genuinely enabled in your framework — vLLM has it on by default in recent versions, while others may need explicit configuration. You also need at least two instances before any of this matters.
If tuning inference economics is already on your list, this sits beside the work in our earlier piece on serving code models with a persistent kernel. Both circle the same underlying question: how much computation can you avoid repeating, and where must the system change to let you avoid it?
The honest limitation is that the gains follow the shape of the workload. RAG over a shared document, multi-turn chat, templated assistants, and code completion all carry long prefixes and should benefit. A workload where every request is genuinely distinct will not. AWS recommends enabling detailed observability so you can track KV cache hit rates at the model level — that is the check that tells you whether this is working for your traffic rather than for someone else’s benchmark.