NVIDIA Dynamo Recipes Put Korean Flagship LLMs on B200s

Summary: On November 8, 2026, NVIDIA released Dynamo-optimized deployment recipes for four of Korea’s leading open models: A.X K2 from SK Telecom, K-EXAONE 2.0 from LG AI Research, Solar Open 2 from Upstage, and Motif-3 from Motif Technologies. The recipes are tuned for B200 GPUs and promise up to 2x performance out of the box. For builders, this means Korea’s flagship open weights now come with tested serving configs instead of a week of hand-tuning multi-GPU inference.

Why serving these models is hard in the first place

The models in this batch are big — big enough that no single GPU holds them. Once you split a model across GPUs, serving stops being a single-process problem: request routing, KV transfer between prefill and decode, and inter-GPU communication all start eating your throughput if you get them wrong.

Dynamo is NVIDIA’s open-source distributed inference-serving framework, built for exactly this. It handles routing and communication across multi-node deployments at data-center scale. The recipes wrap Dynamo’s machinery into deployment configurations you can apply directly, rather than assembling the pieces yourself.

This fits a broader pattern in Korea’s AI infrastructure push, where sovereign compute capacity — like the 250MW AI factory Reflection AI and Shinsegae broke ground on — needs real serving software behind it to matter.

What a recipe actually gives you

“Recipe” sounds vague, so it’s worth looking at what the documentation delivers. The K-EXAONE 2.0 recipe serves LGAI-EXAONE/K-EXAONE-2.0-750B-A37B-NVFP4 — a 764.5B-parameter mixture-of-experts model with roughly 37B active parameters per token, 256 experts at top-8 routing, and hybrid attention: 20 full-attention layers plus 58 sliding-attention layers.

The recipe ships as Kubernetes manifests deployable through the Dynamo Operator, with two fully specified topologies:

Aggregated: 4x B200 on one node, tensor parallelism of 4, NVFP4 W4A4 weights with FP8 KV cache, and Multi-Token Prediction speculative decoding at draft length 2.
Disaggregated (1P1D): 8x B200 split into 4 prefill and 4 decode GPUs, with KV cache transferred over NIXL/UCX via InfiniBand RDMA.

The numbers, and the topology trade-off

The docs benchmark against a Mooncake chat trace replay of 1,805 requests (8K input / 1K output, designed for ~70% prefix reuse), pushed to the highest concurrency meeting a gate of at least 50 tok/s per user and TTFT p50 under 5 seconds. On the aggregated target: 7 concurrency, 87 system output tok/s per GPU, 51.4 user output tok/s (P50), 291 ms TTFT P50, and 19.15 ms ITL p50.

The disaggregated target at 14 concurrency lands at 85 tok/s per GPU, 54.9 user output tok/s (P50), 2,230 ms TTFT P50, and 15.98 ms ITL p50. Read those two rows together and the trade-off is explicit: per-GPU throughput is essentially level with the aggregated target — on twice the GPUs. You pick disaggregation for lower per-token latency and independent prefill/decode scaling, not for throughput. That’s a purchasing decision as much as an architecture one.

Gotchas the docs surface

A few practical frictions are worth planning around before you deploy:

First start takes 40–120 minutes. 53 shards load, then autotune, then CUDA-graph capture. The docs warn that silence during this window is not a hang — watch the worker log for shard progress.
Storage: the checkpoint is roughly 530 GB on disk, and you need a ReadWriteMany storage class with at least 700 GB free.
A correctness pin, not a preference: the manifests force the FLASHINFER_CUTLASS MoE backend. Per the docs, vLLM’s auto-selection picks FLASHINFER_TRTLLM, which silently corrupts long-form output on this checkpoint — fluent, plausible, wrong, with no error raised. If you strip the manifests down, keep this pin.
Prefix reuse underperforms here. The trace is designed for ~70% KV reuse, but measured reuse was 8.8%: the workload’s working set exceeds TP4’s KV capacity, so blocks get evicted before reuse. Expect different cache behavior on your own traffic.
Context ceiling: 38 of the trace’s 1,805 requests exceed the 262,144-token context window and return HTTP 400; they’re excluded from the metrics.
KV-aware routing is wired end to end but inert at the shipped replica count of 1 — it only earns its keep once you scale out.

One serving detail: K-EXAONE 2.0 is a reasoning model. The docs recommend temperature 0.6 over greedy decoding, a generous token budget, and reading .content with a fallback to .reasoning_content — reasoning burns the token budget before the final answer, so a short cap prints null on a perfectly healthy deployment.

What to check next

If you’re bringing one of these four models into production, the recipe removes the config-search phase: pick a topology based on whether you need latency (disaggregated) or GPU efficiency (aggregated), provision the storage and RDMA prerequisites, and apply the manifests. Then replay your own trace with AIPerf — the Mooncake numbers describe a specific workload, not yours.

Also keep scope in mind. The published performance figures cover K-EXAONE 2.0 only; the docs don’t give equivalent numbers for A.X K2, Solar Open 2, or Motif-3, and NVIDIA’s “up to 2x performance” claim isn’t independently verified here. Treat it as a starting point to measure against, not a guarantee.

Sources

Original post

View the original post on X