Most LLM serving stacks treat each decode step as a sequence of separate kernels: launch QKV, wait; launch attention, wait; launch the MoE, wait. Each launch is fine on its own. The problem is the waiting in between. At small batch sizes, the GPU spends a surprising fraction of every decode step waiting rather than computing.
Cohere’s new serving engine for North Mini Code attacks that idle time directly. Instead of a hundred small kernels, it runs the entire forward pass as one persistent megakernel. The result: BF16 on a single H100, 1.25×–1.41× faster than vLLM end-to-end, with no measurable loss of accuracy.
Why decode is memory-bound, not compute-bound
Autoregressive decoding at low batch sizes is fundamentally limited by memory bandwidth. For every decode step, you move a large fraction of weights from HBM while doing relatively little compute. North Mini Code is a 30B model with 3.3B parameters active per token. In BF16, that means streaming 6.6 GB of weights during every decode step, plus roughly 0.5 GB of KV cache at 8K context.
An H100 delivers 3.35 TB/s of HBM bandwidth, putting the Speed-of-Light (SoL) at about 470 tokens per second. vLLM serves this model at 185 tok/s — merely 39% of SoL. The megakernel reaches 292 tok/s at batch size 1, or 62% of SoL, a 1.58× speedup over vLLM. That margin holds across batch sizes and out to 256K of context.
Where the speedup comes from
A kernel-per-operation approach has several places where weights stop moving. Together those stalls account for most of the 61% of bandwidth that a typical inference engine leaves unused.
Reduced launch and synchronization overhead. Between two consecutive kernels, every SM must finish before any SM can start the next one, and the driver has to dispatch the next grid. For a decode step made of dozens of small kernels per layer, those gaps add up. A megakernel pays that cost once per step rather than once per operation.
Reduced wave quantization. Suppose a kernel has 200 tiles of work and the GPU has 132 SMs. The first 132 tiles run in parallel; the remaining 68 run in a second wave while 64 SMs sit idle. The kernel takes two waves’ worth of time to do 1.5 waves’ worth of work. In a megakernel there is no boundary to round up to: a tile whose inputs are ready starts on whichever SM is free.
Dropped false dependencies. SMs do not always finish work at the same time. A kernel boundary is a full-grid barrier, so the slowest SM sets the pace for all of them. Fine-grained barriers drop the false dependency: O-proj for a given KV group starts as soon as that group’s attention output lands, without waiting for all groups.
Weight prefetch. Weights are immutable. A task can start streaming its weight tiles from HBM into shared memory before its activation dependency has been satisfied. Cohere uses this most aggressively for the router and QKV projections, which prefetch their weights during the tail of the previous layer’s O-proj, before RMSNorm has even run.
What a real serving system needs
This is not a standalone demo that measures decode speed at batch size 1. Cohere presents what they believe is the first fully fledged serving system built around a decode megakernel. It supports continuous batching, paged attention, and ragged sequence lengths, all behind an OpenAI-compatible endpoint with tool calling. Point OpenCode at it and you can code with it.
The implementation is a single CUDA file: no compiler, no new programming paradigm, no exotic abstractions — just ordinary tiled GEMMs and ordinary paged attention, restructured to fit a single calling convention. Cohere found that megakernels are much easier to write than their reputation suggests, and they include a recipe for porting kernels you already have into one.
What this changes for builders
If you serve small-batch decode workloads — coding assistants, agentic tools, interactive chat — the gap between 39% and 62% of memory bandwidth is real money. It means fewer GPUs for the same throughput, or lower latency for the same hardware. The megakernel approach is not a research curiosity; it is a production serving engine with an OpenAI-compatible API.
For teams already running small language models in production, the lesson is broader: the bottleneck is not always the model size or the kernel speed. Sometimes it is the scheduling and synchronization between kernels. A persistent kernel that keeps weights moving can beat a stack of individually optimized kernels.
Cohere has open-sourced the serving engine on GitHub. The code is there to inspect, adapt, and measure against your own workloads.
Sources
AI-assisted summary compiled from the sources above, reviewed by a human before publishing.