An inference pod on Amazon SageMaker HyperPod doesn’t serve traffic the moment the scheduler places it. Two sequential downloads sit in front of it: the inference server container image from Amazon ECR, and the model weights from S3, FSx for Lustre, or HuggingFace Hub. The AWS Machine Learning Blog puts numbers on it — 5–7 minutes for a vLLM or LMI image, another 20+ minutes for a 145 GB model on S3, and upwards of 30 minutes for something like DeepSeek-R1 at 600+ GB.
The part that hurts isn’t the first deploy. It’s scale-out. If a HorizontalPodAutoscaler asks for five new pods during a traffic spike, all five repeat the same download sequence independently. The autoscaler reacts in seconds; the capacity arrives 25–30 minutes later. Your scaling policy and your actual serving capacity are running on different clocks.
Two caches, enabled separately
Model caching, launched September 10, 2026, pre-loads data onto nodes before pods are scheduled. It splits into two independent capabilities you can turn on together or alone.
The weights cache downloads model weights to local NVMe on each target node ahead of time. The HyperPod Inference Operator creates a ModelDataCacheConfig resource, downloads from your configured source, labels each node cache-ready when its download finishes, and waits for all target nodes before creating the inference deployment. Pods then read locally at roughly 7 GB/s. The cache survives pod restarts on the same node.
The image cache pre-pulls the container image via a DaemonSet. Unlike the weights cache, it does not block deployment creation — a pod that starts before the pull completes just pulls from ECR as usual. Multiple deployments sharing one image share a single cache resource, and the operator only cleans it up when nothing references it.
Fallback is preferred, not required
Both caches use preferred scheduling. Pods prefer warm nodes but are never blocked from starting on a cold one. During rapid scale-out past the number of cached nodes, the pod falls back to the original S3 or FSx source and a normal ECR pull. No failure, no intervention — just the download time you had before.
That design choice matters more than the benchmark numbers. It means enabling caching can’t make a deployment less reliable than it already was.
What the benchmarks actually claim
Across models from 57–145 GB, AWS reports around 60 percent faster scale-out with weights caching. The image cache removes over two minutes of cold pull time, described as up to a 97 percent reduction versus pulling fresh from ECR on every pod start. The benefit scales with model size, since there’s proportionally more data to move.
Treat these as vendor benchmarks on unspecified hardware and network conditions. The direction is credible — local NVMe beats network reads — but your own numbers will depend on instance type, storage backend, and how much of your traffic is genuinely bursty.
The constraints that decide your instance type
Weights caching is per-node. Every node keeps its own copy, so NVMe consumption scales with node count, not model count. If your model is 300 GB and your instance has 250 GB of NVMe, caching simply won’t work. AWS lists ml.g5.xlarge at 250 GB up to ml.p5.48xlarge at 30,000 GB, so the sizing question comes before the config question.
Two more limits worth planning around. The first cache population still pays the full remote download once. And source updates aren’t auto-detected: if you overwrite model files at the same S3 path without changing the spec, the operator keeps serving the cached version. You need to change the model path or add a version suffix to pick up new weights.
Where this fits next to routing work
Caching solves the cold-node problem. It doesn’t solve the warm-node problem — what happens when several pods are already serving and a request lands on the one holding the wrong KV cache. That’s a separate lever, and we looked at it in prefix-aware routing on SageMaker. The two compose: caching gets capacity online faster, routing decides which online capacity handles a given prompt.
Enabling it is a modelCacheConfig block on an existing InferenceEndpointConfig or JumpStartModel resource — no extra infrastructure. Before you flip it on, check NVMe headroom against your largest model and confirm you have a versioning habit for weight updates. Otherwise you’ll cache something and quietly keep serving it.
Sources
AI-assisted summary compiled from the sources above, reviewed by a human before publishing.