Summary: AWS launched model caching for SageMaker HyperPod on September 10, 2026 to pre-place model weights and container images on node NVMe. Weights caching blocks deployment until target nodes are cache-ready, while image caching pre-pulls images without blocking. AWS reports 60% faster scale-out for 57–145 GB models and vendor benchmarks showing up to 97% less image-pull time.
Getting an inference pod scheduled onto Amazon SageMaker HyperPod is not the same thing as getting it to serve traffic. Two downloads run in sequence before that can happen: the inference server container image comes from Amazon ECR, and the model weights come from S3, FSx for Lustre, or HuggingFace Hub. The AWS Machine Learning Blog spells out the cost — 5–7 minutes for a vLLM or LMI image, another 20+ minutes for a 145 GB model on S3, and upwards of 30 minutes for a model like DeepSeek-R1 at 600+ GB.
The expensive part isn’t the first deploy; it’s scale-out. When a HorizontalPodAutoscaler asks for five new pods during a traffic spike, each of the five walks the same download sequence on its own. The autoscaler reacts in seconds, the capacity shows up 25–30 minutes later — your scaling policy and your actual serving capacity are keeping different clocks.
Two caches you can enable independently
Launched September 10, 2026, model caching puts data on the nodes ahead of pod scheduling. It arrives as two independent capabilities, which you can switch on together or one at a time.
With the weights cache, model weights are downloaded to local NVMe on every target node ahead of time. The HyperPod Inference Operator creates a ModelDataCacheConfig resource, pulls from whichever source you configured, marks each node cache-ready once its download completes, and waits until all target nodes reach that state before creating the inference deployment. Pods then read from local storage at roughly 7 GB/s, and the cache outlives pod restarts on the same node.
The image cache pre-pulls the container image through a DaemonSet. Where it differs from the weights cache is that it never blocks deployment creation — a pod that starts before the pull finishes simply pulls from ECR the way it always did. When several deployments share an image they also share one cache resource, and the operator only cleans that resource up once nothing references it anymore.
Fallback is preferred, never required
Preferred scheduling is what both caches rely on. A pod would rather land on a warm node, but a cold one never prevents it from starting. When rapid scale-out goes past the number of cached nodes, that pod falls back to the original S3 or FSx source and to a standard ECR pull. Nothing fails, nobody has to intervene — you just pay the download time you paid before.
That decision carries more weight than any benchmark figure: turning caching on cannot make a deployment less reliable than it already was.
What the benchmarks do and don’t claim
On models ranging from 57–145 GB, AWS reports scale-out roughly 60 percent faster with weights caching enabled. The image cache strips out over two minutes of cold pull time, described as up to a 97 percent reduction compared with pulling fresh from ECR at every pod start. The gain grows with model size, simply because there is proportionally more data to move.
Worth treating as vendor benchmarks, since the hardware and network conditions behind them go unspecified. The direction holds up — reading from local NVMe beats reading over the network — but the figures you get will depend on your instance type, your storage backend, and how much of your traffic is genuinely bursty.
The constraints that pick your instance type for you
Caching weights happens per node. Every node holds its own copy, so NVMe use grows with node count rather than model count. A 300 GB model on an instance with 250 GB of NVMe is a non-starter for caching. AWS’s range runs from 250 GB on ml.g5.xlarge up to 30,000 GB on ml.p5.48xlarge, which makes sizing a question that comes before configuration.
Two further limits deserve a place in your planning. The initial population of a cache still pays the full remote download once. And updates at the source are not picked up automatically: overwrite model files at the same S3 path without touching the spec and the operator goes on serving the cached version. Picking up new weights requires changing the model path or appending a version suffix.
How this sits alongside routing work
What caching fixes is the cold-node problem. What it leaves alone is the warm-node problem — the case where several pods are already serving and a request lands on the one holding the wrong KV cache. That needs a different lever, and we looked at it in prefix-aware routing on SageMaker. The two compose: caching brings capacity online sooner, routing decides which of the online capacity handles a given prompt.
Turning it on means adding a modelCacheConfig block to an existing InferenceEndpointConfig or JumpStartModel resource — no extra infrastructure is involved. Before you do, compare NVMe headroom against your largest model and make sure you have a versioning habit for weight updates. Otherwise you’ll cache something and quietly keep serving it.