SageMaker GPU Partitioning: Carve a GPU Seven Ways

Summary: Amazon SageMaker Inference now supports GPU partitioning built on NVIDIA Multi-Instance GPU (MIG), so a real-time endpoint can split each physical GPU into up to seven hardware-isolated partitions and place a different model on each. AWS announced the feature on October 30, 2026 and published a detailed walkthrough on the AWS Machine Learning Blog on November 6, 2026. If you serve many small or mid-sized models — fine-tuned variants, embeddings, rerankers, classifiers — this changes how many models fit on one instance and how precisely you can size what each one gets.

The constraint that made you waste GPU

Until now, the smallest unit of GPU you could request on SageMaker was one whole accelerator. That hurt twice. An ml.p4d.24xlarge has eight A100s, so it hosted at most eight models, whether each needed 40 GB or 14 GB. And as concurrency and prompt lengths grew, you were stuck between a full older-generation GPU that stopped being enough and a full newer-generation GPU your model could never saturate.

The existing workarounds each fell short, per the AWS post. Multi-model endpoints share a GPU without isolation, so one model’s heavy request hits the others’ latency, and unloaded models pay a first-use load penalty. Customers who genuinely needed partitioning moved to self-managed Kubernetes, where MIG already worked — but that meant owning driver versioning, partition configuration, a device plugin, and fleet-wide consistency. MIG support in SageMaker removes the platform work: you declare the partitioning, and SageMaker handles GPU configuration, placement, and scaling.

Two optional fields, no new APIs

The mechanism is deliberately small. Partitioning adds two optional fields to calls you already make — no new API operations, resource types, or IAM actions — and an endpoint that omits both behaves exactly as before. It builds on inference components, the construct for hosting several models on one endpoint; if you use single-model endpoints or serverless endpoints, partitioning does not apply.

AcceleratorPartitionLayouts goes on a production variant in CreateEndpointConfig and declares how each accelerator is divided. AcceleratorPartitionType goes on an inference component and selects which slice the model runs on:

{
  "ProductionVariants": [{
    "VariantName": "AllTraffic",
    "InstanceType": "ml.p4d.24xlarge",
    "AcceleratorPartitionLayouts": [
      { "AcceleratorPartitionTypes": ["mig-4g.20gb", "mig-2g.10gb", "mig-1g.5gb"] }
    ]
  ]
}

{
  "InferenceComponentName": "summarizer",
  "Specification": {
    "ComputeResourceRequirements": {
      "AcceleratorPartitionType": "mig-4g.20gb",
      "NumberOfAcceleratorDevicesRequired": 1
    }
  }
}

One layout entry describes one accelerator, and every accelerator on every instance in the variant gets the same carve. With that 4g+2g+1g layout on a p4d, each of the eight A100s carries three partitions, so the endpoint offers 24 in total — eight of each type. Raising CopyCount on a component places extra copies on free partitions, which helps a small model under high concurrency where one process leaves the GPU idle. Invocation doesn’t change: InvokeEndpoint still addresses a model by component name, and partitions never appear in the request.

The profile math, and one trap

Profiles are a property of the GPU, and the naming differs between families. On P4d (A100, 40 GB) and P5 (H100, 80 GB) you get up to 7 partitions; on G7 (RTX PRO 4500 Blackwell, 32 GB) only 2; on G7e (RTX PRO 6000 Blackwell Server Edition, 96 GB) up to 4. Compute divides in sevenths and memory in eighths on the A100/H100, in halves on G7, in quarters on G7e. Validation happens at endpoint creation, and an invalid layout is a rejection, not a fallback to a whole GPU.

Two details worth memorizing. The mig-3g.20gb profile takes three of seven compute units but four of eight memory units — two of them fill the whole device and nothing fits alongside. And mig-1g.10gb on P4d needs an R525-or-newer NVIDIA driver, while the default P4d image ships 470.256.02; requesting it there fails without an error naming the profile. The fix AWS gives is setting InferenceAmiVersion to al2023-ami-sagemaker-inference-gpu-4-1 on the production variant to get a 580-line driver.

Because a partition hands you compute and memory in fixed proportions, AWS frames layout choice around inference phases: compute units drive prefill (first-token latency on long inputs), while memory governs decode and model capacity. The blog post’s guidance on matching layouts to specific workloads is cut off at the end of the captured text, so treat that section as incomplete rather than the author’s final word.

What it costs and what it rules out

Partitioning happens during endpoint provisioning, and AWS reports that “carving accelerators added roughly 17 seconds to provisioning” in their testing — on top of normal instance launch and container pull. The bigger operational limit: MIG only allows layout changes while partitions are idle, so on SageMaker your layout is fixed at endpoint creation. Changing it means a new endpoint. A model needing more than one GPU still needs whole devices.

The sources describe no benchmark results on cost savings, utilization gains, or latency beyond that provisioning overhead. Salesforce’s Richard Chang says the company helped define requirements and validated the initial solution, and is now preparing to validate with representative production workloads — that’s intent, not yet production evidence.

What to do next

Availability is limited to P4d, P5, G7, and G7e in seven regions, including US East (N. Virginia), Asia Pacific (Tokyo), and Europe (Frankfurt), per the October 30 announcement. Check your region first. Then, before migrating anything, size your fleet honestly: profile each model’s prefill compute and weight memory, pick profiles per NVIDIA’s supported MIG documentation, and verify your layout against the compute-and-memory unit limits above. If you’re tracking this alongside SageMaker’s other recent serving changes, our earlier 2026 SageMaker inference launch roundup covers the rest of the stack. And if a p4d carve fails silently, check the driver note before anything else — that failure mode names nothing.

Sources

Original post

View the original post on X