AGENTIC COMMONSAI industry briefings

繁中EN

TagBenchmarkingPublished

All English articles

OpenRouter's Benchmark Recipe Starts With Your Own Tasks

Summary: OpenRouter published a hands-on tutorial (How to Benchmark LLMs, 11/12/2026) on grading models against tasks with answers a script can check, rather than trusting public leaderboards. It ships a runnable Python package that keeps offline validation free of inference calls, requires an explicit --live flag and a --max-usd allowance before any paid run, and reconciles costs against verified bills before a report resolves them. On OpenRouter’s own 60 synthetic tasks, GPT-6.1 Sol passed all 180 trials — but the resulting ranking barely tracked public leaderboards.

What the method actually does

Each task has an input, an expected answer, and an output contract. OpenRouter’s examples cover invoice arithmetic with decimal totals, timestamp conversion, extracting email fields into JSON, implementing Python functions against hidden inputs inside an isolated runtime, and applying a refund policy at its boundaries. Grading is code: exact matching for tokens like APPROVE, schema and value checks for JSON, with normalization limited to differences you actually permit — the tutorial notes that accepting prose around a number hides a failure when your consumer needs only the number.

The step most teams skip is testing the grader itself. OpenRouter’s fixtures cover known-correct, wrong, and malformed responses, plus boundary values like exactly $200 and 30 days with values just above each threshold. Offline checks run with zero inference calls; live runs only happen after an explicit --live flag and a dollar cap, and a report keeps its cost figure unresolved until reconciliation against verified bills confirms it.

The numbers, and the one that matters

OpenRouter wrote 40 baseline and 20 hard tasks before the final run, with three repeats per task. Under the tested contracts and settings, GPT-6.1 Sol passed 180/180 trials; DeepSeek had the lowest response cost among the tested models, with 157/180 passes. The striking result is agreement: OpenRouter’s task-specific ranking correlated with the public leaderboard at a Spearman of just 0.109.

The price of trying this yourself is small. OpenRouter’s planning output estimated about $0.0024 for a three-task run on its Luna configuration and about $0.047 on Sol, with price ceilings set in USD per million tokens — Luna’s prompt ceiling is roughly $0.10. These reservations are local estimates, not account holds; if spending exceeds the estimate, the script stops sending new requests.

Your grader is part of the measurement

OpenRouter’s own development checker initially rejected valid Python because it omitted ord and type. It was corrected before the final study, pinning CPython 3.12.10 on WASI with resource limits — otherwise, as the tutorial puts it, checker failures would have counted against models. That’s a concrete warning: a buggy grader silently distorts a benchmark in whatever direction its bug points.

Platform support is narrow for the heavyweight parts. The bundled full-study sandbox and native Ori setup support Windows x64 only; macOS remains untested. The teaching runner passed offline checks on Windows x64 and Linux. The full tutorial goes further on custom tasks and reporting than we cover here.

Our take: Act now, but act small. The number that should change behavior is 0.109: if leaderboard order barely tracks OpenRouter’s own tasks, our assumption is it will track your structured-output workload even less. If you’re a small team picking between models for invoice math, date handling, or policy decisions and currently choosing names off a leaderboard, copy this recipe this week — the three-task paid run costs a few cents on the Luna ceilings, and nothing is locked in. Teams with stable, measured results already in production, or anyone not weighing a model change, can safely ignore this. Two cautions before you generalize: these are OpenRouter’s own synthetic tasks under specific contracts, so a reason to run a test, not a verdict on Sol or DeepSeek. And your grader is now part of the score — OpenRouter’s checker bug shows the downside. We’d reverse this recommendation if your own runs reproduce the leaderboard order, or if independent replications show their 60-task set predicts real workloads well enough to skip writing your own. For the task set itself, pairing it with real traffic — see our earlier piece on turning production traffic into an eval set — beats pure synthetic cases.

Sources

AGENTIC COMMONSOperated by PHLEGON LABS
ShareXEmail
Support us

Related reading

  1. OpenRouter's Image Benchmarks: A Practical Guide for Product Builders

    OpenRouter launches visual image benchmarks to help developers compare 39 image models across 7 challenge categories, with practical advice for product builders.

    Image Generation

  2. Open-Weight Models Hit 72% of Tokens on OpenRouter

    OpenRouter data shows open-weight models now handle 72% of top-50 tokens at roughly a quarter of closed-model prices.

    OpenRouter

  3. A Leaderboard for the Layer Above Your Model

    OpenRouter now benchmarks model routers on quality, speed, and cost, so you can decide when routing beats picking one model.

    OpenRouter