Nothing kills a voice agent like silence. If a user stares at a screen for three hundred milliseconds before hearing anything, the experience is already broken. On August 19, Nari Labs published a research post showing how far the serving layer can move that number: their Qwen3-TTS 1.7B deployment sustains 10 requests per second with a p95 time-to-first-audio (TTFA) under 50 ms on a single NVIDIA H100 SXM, without dropping playback. Both the implementation and the benchmark are open source.
What “Real-Time” Speech Actually Requires
Nari breaks real-time TTS serving into four requirements. First audio must arrive fast: the time from request dispatch to the first audible sample needs to be low. Playback must not run dry: the client buffer must never underrun. Capacity must hold: both properties need to survive rising request rates. And output must be well-formed: the speech has to be intelligible.
The model is Qwen3-TTS 1.7B CustomVoice, chosen for practical reasons — it is one of the most popular TTS models with a permissive license. The measurement discipline matters as much as the target: each run lasts five minutes under Poisson open-loop traffic, following Fireworks AI’s LLM benchmark methodology. The team detects the audible TTFA, reconstructs playback from received PCM, and evaluates completed audio with Deepgram speech-to-text. Five engines, one ruler.
Five Engines, One Benchmark
Nari compared their implementation against vLLM-Omni, SGLang-Omni, VoxServe, and M* — and, importantly, tuned every engine for low-latency streaming before comparing. The results are unforgiving. After tuning, only VoxServe clears the bar at 1 RPS, with a p95 TTFA of 49.3 ms; vLLM-Omni lands at 56.8 ms, while M* (104.0 ms) and SGLang-Omni (120.9 ms) trail well behind. By around 6 RPS, all four baseline engines sit at roughly 100 ms or higher.
Nari’s version holds sub-50 ms p95 TTFA all the way to 10 RPS and stays under 100 ms even at 20 RPS. Same model, same hardware — the difference is entirely serving-layer engineering.
One Scheduler for Three Modules
Qwen3-TTS is a three-part model doing hierarchical multi-codebook generation. The Talker predicts the first codebook token for each audio frame, the Code Predictor generates the remaining 15 codebook tokens, and a causal Codec turns tokens into waveform samples. Most serving stacks run the Talker and Code Predictor together and schedule the Codec separately.
Nari goes the other way: all three modules become independently schedulable tasks under one shared scheduler, a design that draws on M*. The key insight is not the splitting but the unification. Merging the Talker and Code Predictor looks efficient because it removes an intermediate boundary, but the combined operation becomes a non-preemptible unit of work that blocks more urgent Codec jobs. Keeping modules separate creates shorter units of work and gives the scheduler room to interleave requests by urgency instead of following a fixed execution order.
Scheduling Around Speech Urgency
Speech streaming has two fundamentally different notions of urgent. Before the first chunk of audio arrives, every millisecond adds directly to perceived latency. But once playback starts, the next chunk only needs to land before the current audio finishes playing — arriving earlier gives the user nothing.
So Nari’s scheduler gives high priority to requests that have not yet produced audio, while established streams become urgent only as they approach their playback deadline. To protect batching efficiency, the scheduler picks a single urgent request as an anchor and fills the rest of the batch with compatible work: urgent jobs jump the queue, but batches still leave the GPU full.
What It Means for Voice Developers
Three signals for anyone building voice products. First, the competitive frontier for TTS is shifting from audio quality to latency and concurrency: with the model held constant, serving-layer engineering moves the numbers by an order of magnitude, so the questions to ask a vendor are about their scheduler, not their vocoder. Second, write “real-time” into your SLOs — p95 first-audio latency, underrun counts, and behavior under load say more than any demo. Third, the implementation and benchmark are open source, so teams running their own voice stack can measure the technique against their real workload instead of guessing from a website demo. The sub-50 ms bar is now public, reproducible, and yours to test against.
Sources
AI-assisted summary compiled from the sources above, reviewed by a human before publishing.