Generative AI inference is awkward in ways ordinary web serving is not. Weights run into the tens or hundreds of gigabytes, latency is measured in tokens per second, cold starts can stretch across minutes while containers and weights move, and GPU capacity is not something you can order on demand. The AWS Machine Learning Blog’s year-to-date review of SageMaker inference launches, published September 18, 2026, walks through 13 capabilities shipped across two deployment paths in 2026. The useful question for a builder is not how many features landed, but which ones remove a decision you were making badly.
Two paths, and the choice you actually make
The post frames the split clearly. Managed endpoints are for teams that want AWS to handle GPU provisioning, scaling, and operational monitoring. HyperPod Inference is for teams that need Kubernetes-native control over dedicated GPU clusters, with node-level access and their own frameworks and AMIs. Endpoints scale through managed auto scaling with CloudWatch; HyperPod scales through Karpenter, KEDA, and CloudWatch. Endpoints speak an OpenAI-compatible protocol; HyperPod offers HTTP, gRPC, and custom load balancer capability.
That is a real fork, not a marketing one. If your team already runs Kubernetes and wants train-to-serve continuity or hybrid deployment, HyperPod is the fit. If you want the model in production without owning the cluster, endpoints are the shorter road.
The launches that change day-to-day serving
Several of the 2026 additions target problems that used to cost weeks or cause outages.
Inference recommendations (April 2026) automates instance, container, and optimization selection that the post says typically takes two to three weeks of manual benchmarking against more than 1,000 combinations. You specify a model and a goal — cost, latency, or throughput — and SageMaker narrows the instance space, applies techniques like speculative decoding or tensor parallelism, then benchmarks on real GPU infrastructure. The output is a Model Package with validated TTFT, ITL, percentile latency, throughput, and cost projections. In the demonstrated example, throughput optimization on GPT-OSS-20B delivered 2x tokens per second at the same request latency. There is no additional cost for generating recommendations.
Capacity-aware instance pools (May 2026) attack a single point of failure: when an endpoint needed one instance type and capacity ran out, it failed before serving a request. You now define a prioritized list of up to five instance types, and SageMaker walks that list at creation, during scale-out, and during scale-in — removing fallback instances first so the fleet drifts back toward preferred hardware. Each pool entry can carry its own optimized model configuration.
Container caching (June 2026) removes the image pull from scale-out. For serving containers over 10 GB, that pull added minutes of dead time to every new instance. With Qwen3-8B on ml.g6.2xlarge using the LMI container, the post reports end-to-end startup latency dropping from 525 seconds to 258 seconds, and model download time from 168 to 77 seconds. It requires no configuration or container changes.
OpenAI-compatible APIs (May 2026) cut migration cost for anything built on the OpenAI SDK, LangChain, or Strands Agents: change the endpoint URL, keep the SDK calls, streaming logic, and prompt formatting. Bearer tokens generated from existing AWS credentials replace SigV4 signing.
Async inference inline payloads (June 2026) let InvokeEndpointAsync accept a Body parameter up to 128,000 bytes, removing the S3 pre-staging step for most async workloads — one fewer round trip, no input bucket or s3:PutObject grant, and no S3 PUT charge per invocation.
Observability is the launch to read twice
The June 2026 observability release may matter more than any single performance feature. SageMaker now emits 100+ detailed inference metrics through native OpenTelemetry, with a pre-built CloudWatch Insights dashboard and no instrumentation. New endpoints have it on by default, with metrics within two minutes of reaching InService. The dashboard covers performance (TTFT, ITL, throughput, KV cache utilization, queue depth), capacity (GPU utilization, memory, temperature, disk), and reliability (Availability Zone distribution with risk scoring, cold start anatomy, scaling event history). A PromQL-compatible endpoint lets you query from Amazon Managed Grafana.
Token-level signals are exactly what generic monitoring misses. If you have been debugging inference incidents by correlating scattered metrics after users complain, this is the change that shortens that loop. It also pairs naturally with the kind of cache-placement thinking we wrote about in Kimi K3 on Bedrock: where the cache breakpoint goes — knowing where latency actually accumulates is the prerequisite for fixing it.
What to do with this
If you are choosing a path this quarter, start with inference recommendations rather than a spreadsheet of instance types; the benchmarking is free and the output is deployable. If you already run endpoints, check whether capacity-aware pools and container caching are enabled, since both reduce failure modes you may be absorbing manually. The supplied review covers 13 launches in total, and the post itself is the place to read the remaining ones — including the HyperPod-side work on tiered KV cache, disaggregated prefill and decode, and model caching — before deciding what your stack needs next.
Sources
AI-assisted summary compiled from the sources above, reviewed by a human before publishing.