Descript builds Underlord, a video editing agent. Testing a new model used to take a couple of hours of engineering work and about a week of waiting, because the person who wanted to try a model was not the person who could add it. According to OpenRouter’s case study writeup, the team maintained three direct integrations (OpenAI, Anthropic, Google) plus its own fallback logic. Adding a model meant finding an engineer’s calendar slot first.
The expensive part was never the two hours of wiring. It was the queue. Descript tried the models worth interrupting an engineer for and skipped the rest.
What the loop looks like now
Aleks Mistratov, Head of AI Product at Descript, describes the new path in the case study: he sees a model release, posts in Slack asking for evals on OpenRouter with a link, and an hour or two later the evals have run in Descript’s own harness. Claude Tag, Anthropic’s Slack integration, runs the evaluations and opens pull requests; a human reviews and approves.
The mechanism matters less than the precondition. Trying a model no longer requires building a provider integration first, so testing does not start with a vendor relationship or a new API endpoint. There is still work in the loop. Nobody has to be asked for it.
Three consequences of evaluating weekly
Descript now evaluates models multiple times a week, and most of what it tests does not ship. The case study names three effects:
- Model choice becomes empirical. New releases are measured against Descript’s own test cases, and the shipped model changes when something clears the bar. The prior alternative was standardizing early and revisiting only under pressure.
- Releases stop being events. On a recent Anthropic release, Descript went from announcement to live in a couple of hours: run evals, decide whether the model beats the incumbent, add it to the model picker, deploy. Most of that remaining work lives in Descript’s own code structure, not in the model connection.
- The field widens. Descript never built a direct xAI integration, yet Grok 4.5 runs in production, added through the same path as everything else.
Testing and running are not the same configuration
The case study is explicit that this is not “route everything through one endpoint and accept whatever comes back.” Descript brings its own key to Baseten for a dedicated deployment of an open-weight model, with OpenRouter routing to Baseten first and other providers as fallbacks behind it. Those fallbacks swap the inference provider while keeping the model constant, so a host failure does not silently change which model answers.
That distinction is the part worth copying. The same integration serves two jobs: testing is how the team finds a model, routing configuration is how it runs one. Collapsing those into a single default is where teams lose control of what actually answered.
What this changes for your own eval setup
If your evaluation bottleneck is a person rather than a harness, adding more test cases will not help. The queue is the constraint. Descript’s fix was to make model access a parameter instead of a project, which is the same instinct behind giving your coding agent deployment facts rather than a model name: the useful abstraction is the one that lets a decision be made without a new integration each time.
Mistratov’s advice to another engineer weighing the change is short, per the case study: it is far easier than figuring out all the different connections, and you do it once so everything else is a parameter.
Two limits are worth stating. The supplied writeup does not give Descript’s eval suite size, scoring method, or how it decides a model “clears the bar” — those details live in the linked case study, not in this summary. And the loop still ends with a human reviewing pull requests, which is a deliberate choice, not an oversight. If you want to test the pattern, the smallest version is picking one recurring model question, wiring a single access path, and timing how long the answer takes today.
Sources
AI-assisted summary compiled from the sources above, reviewed by a human before publishing.