Summary: Cohere introduced a serving engine for North Mini Code that runs the full decode forward pass as one persistent megakernel. In BF16 on a single H100, it reports 1.25× to 1.41× faster end-to-end performance than vLLM without measurable accuracy loss. The engine supports continuous batching, paged attention, ragged sequences, and OpenAI-compatible tool calling, and its code is open-sourced.
Most LLM serving stacks execute a decode step as a chain of separate kernels: launch QKV, wait; launch attention, wait; launch the MoE, wait. Any single launch is cheap on its own. What costs you is the gap between them. When batch sizes are small, the GPU burns a striking share of every decode step sitting idle rather than computing.
Cohere’s new serving engine for North Mini Code goes straight at that idle time. Rather than issuing a hundred small kernels, it executes the entire forward pass as one persistent megakernel. The payoff: BF16 on a single H100, 1.25×–1.41× faster than vLLM end-to-end, with no measurable loss of accuracy.
Why decode is memory-bound, not compute-bound
At low batch sizes, autoregressive decoding is bound by memory bandwidth before anything else. Each decode step drags a large share of the weights out of HBM while doing comparatively little arithmetic. North Mini Code carries 30B parameters, with 3.3B active per token. In BF16 that works out to streaming 6.6 GB of weights on every decode step, plus roughly 0.5 GB of KV cache at 8K context.
An H100 moves 3.35 TB/s from HBM, which puts the Speed-of-Light (SoL) near 470 tokens per second. vLLM serves this model at 185 tok/s — just 39% of SoL. The megakernel reaches 292 tok/s at batch size 1, or 62% of SoL, a 1.58× speedup over vLLM. That margin survives across batch sizes and out to 256K of context.
Where the speedup comes from
Under a kernel-per-operation design there are several points where the weights simply stop moving. Summed up, those stalls explain most of the 61% of bandwidth that a typical inference engine never uses.
Launch and synchronization overhead, paid once. Between two consecutive kernels, every SM must finish before any SM may begin the next, and the driver has to dispatch a fresh grid. Across a decode step built from dozens of small kernels per layer, those gaps accumulate. A megakernel pays that price once per step instead of once per operation.
Wave quantization, removed. Take a kernel with 200 tiles of work and a GPU with 132 SMs. The first 132 tiles run in parallel; the remaining 68 run in a second wave while 64 SMs sit idle. The kernel therefore spends two waves’ worth of time on 1.5 waves’ worth of work. A megakernel has no boundary to round up to: a tile whose inputs are ready begins on whichever SM happens to be free.
False dependencies, dropped. SMs do not all finish at the same moment. A kernel boundary is a full-grid barrier, so the slowest SM sets the pace for everyone. Fine-grained barriers drop the false dependency: O-proj for a given KV group starts the instant that group’s attention output lands, without waiting on the other groups.
Weight prefetch. Weights are immutable. A task may begin streaming its weight tiles from HBM into shared memory before its activation dependency has been satisfied. Cohere leans on this hardest for the router and QKV projections, prefetching their weights during the tail of the previous layer’s O-proj — before RMSNorm has even run.
What a real serving system needs
This is not a standalone demo that measures decode speed at batch size 1. Cohere presents what it believes to be the first fully fledged serving system built around a decode megakernel. It handles continuous batching, paged attention, and ragged sequence lengths, all behind an OpenAI-compatible endpoint with tool calling. Point OpenCode at it and you can code with it.
The implementation fits in a single CUDA file: no compiler, no new programming paradigm, no exotic abstractions — just ordinary tiled GEMMs and ordinary paged attention, restructured to fit one calling convention. Cohere found megakernels far easier to write than their reputation suggests, and they include a recipe for porting kernels you already have into one.
What this changes for builders
If your workload is small-batch decode — coding assistants, agentic tools, interactive chat — the distance between 39% and 62% of memory bandwidth translates directly into money. Fewer GPUs for the same throughput, or lower latency on the same hardware. The megakernel approach is no research curiosity; it is a production serving engine with an OpenAI-compatible API.
For teams already running small language models in production, the lesson reaches further: the bottleneck is not always the model size or the speed of a kernel. Often it is the scheduling and synchronization between kernels. A persistent kernel that keeps the weights in motion can outperform a stack of individually optimized kernels.
Cohere has open-sourced the serving engine on GitHub. The code is there to inspect, adapt, and measure against your own workloads.