Summary: Nari Labs reported a Qwen3-TTS 1.7B deployment holding 10 requests per second with p95 time-to-first-audio below 50 milliseconds on one NVIDIA H100. Its scheduler treats the Talker, Code Predictor, and Codec as separately schedulable tasks and prioritizes requests awaiting first audio. The open implementation and benchmark show that serving-layer design can move realtime speech results substantially while keeping the model and hardware fixed.
Silence is what breaks a voice agent fastest. A user who stares at the screen for three hundred milliseconds without hearing anything already reads the experience as broken. In a research post published on August 19, Nari Labs showed how far the serving layer can push that number: their Qwen3-TTS 1.7B deployment holds 10 requests per second with a p95 time-to-first-audio (TTFA) below 50 ms on a single NVIDIA H100 SXM, and playback never drops. The implementation and the benchmark are both open source.
The Four Requirements for “Real-Time” Speech
Nari splits real-time TTS serving into four requirements. First audio has to be fast: the span from request dispatch to the first audible sample must stay short. Playback must not run dry: the client buffer can never underrun. Capacity has to hold: both of those properties must survive rising request rates. And the output has to be well-formed: the speech must be intelligible.
The model is Qwen3-TTS 1.7B CustomVoice, picked for practical reasons — it is one of the most popular TTS models and it carries a permissive license. The discipline applied to measurement matters as much as the target. Every run lasts five minutes under Poisson open-loop traffic, following Fireworks AI’s LLM benchmark methodology. The team detects the audible TTFA, reconstructs playback from the PCM it receives, and evaluates the finished audio with Deepgram speech-to-text. Five engines, one ruler.
Five Engines, One Benchmark
Nari put their implementation up against vLLM-Omni, SGLang-Omni, VoxServe, and M* — and, importantly, tuned every engine for low-latency streaming before any comparison began. The results are unforgiving. After tuning, only VoxServe clears the bar at 1 RPS, at a p95 TTFA of 49.3 ms; vLLM-Omni lands at 56.8 ms, while M* (104.0 ms) and SGLang-Omni (120.9 ms) trail well behind. By around 6 RPS, all four baseline engines sit at roughly 100 ms or higher.
Nari’s version holds sub-50 ms p95 TTFA all the way to 10 RPS and stays under 100 ms even at 20 RPS. Same model, same hardware — the difference is entirely serving-layer engineering.
One Scheduler for Three Modules
Qwen3-TTS is a three-part model doing hierarchical multi-codebook generation. For each audio frame the Talker predicts the first codebook token, the Code Predictor generates the remaining 15 codebook tokens, and a causal Codec turns tokens into waveform samples. Most serving stacks run the Talker and Code Predictor together and schedule the Codec separately.
Nari goes the other way: all three modules become independently schedulable tasks under one shared scheduler, a design that draws on M*. The key insight is not the splitting but the unification. Merging the Talker and Code Predictor looks efficient because it removes an intermediate boundary, but the combined operation turns into a non-preemptible unit of work that blocks more urgent Codec jobs. Keeping modules separate creates shorter units of work and gives the scheduler room to interleave requests by urgency rather than follow a fixed execution order.
Scheduling Around Speech Urgency
Speech streaming has two fundamentally different notions of urgent. Until the first chunk of audio arrives, every millisecond adds directly to perceived latency. Once playback starts, though, the next chunk only needs to land before the current audio finishes playing — arriving earlier gives the user nothing.
So Nari’s scheduler gives high priority to requests that have not yet produced audio, while established streams become urgent only as they approach their playback deadline. To protect batching efficiency, the scheduler picks a single urgent request as an anchor and fills the rest of the batch with compatible work: urgent jobs jump the queue, but batches still leave the GPU full.
What It Means for Voice Developers
Three signals for anyone building voice products. First, the competitive frontier for TTS is shifting from audio quality to latency and concurrency: with the model held constant, serving-layer engineering moves the numbers by an order of magnitude, so the questions to ask a vendor are about their scheduler, not their vocoder. Second, write “real-time” into your SLOs — p95 first-audio latency, underrun counts, and behavior under load say more than any demo. Third, the implementation and benchmark are open source, so teams running their own voice stack can measure the technique against their real workload instead of guessing from a website demo. The sub-50 ms bar is now public, reproducible, and yours to test against.