Summary: AutomationBench, a public benchmark Zapier built to test whether AI agents can genuinely carry business work through to completion, went live on April 20, 2026. Its task pool is drawn from real workflow patterns on the company’s own platform — more than 2 billion monthly task runs across 3.7 million companies — and covers sales, marketing, operations, support, finance, and HR.
AutomationBench, a public benchmark Zapier built to test whether AI agents can genuinely carry business work through to completion, went live on April 20, 2026. Its task pool is drawn from real workflow patterns on the company’s own platform — more than 2 billion monthly task runs across 3.7 million companies — and covers sales, marketing, operations, support, finance, and HR. One day later, on April 21, Daniel Shepard and Robin Salimans submitted the companion paper, “AutomationBench,” to arXiv as 2604.18934.
The significance: evaluations for agents usually stop at single-turn answers inside toy setups. AutomationBench instead ties each score to whether the right data actually landed in the right system. The numbers are unforgiving — GPT 6 Astra (Max), the leaderboard’s top entry, finishes only 41.4% of tasks under strict scoring.
47 Simulated Apps, 600+ Tasks
What the benchmark is built from:
- More than 600 held-out tasks; researchers get a public set, while leaderboard results are scored against a separate private held-out set
- Six business functions covered by 47 simulated apps and roughly 500 API endpoints
- Every task boots its own simulated company, complete with CRM records, inboxes, calendars, spreadsheets, and support cases
- Traps are planted on purpose: rows gone stale, names that nearly duplicate one another, policies hidden inside email threads
- A single trigger message sets the agent off; it then runs with no human in the loop, capped at 50 steps per task
The paper makes a point of this: agents have to locate endpoints on their own, honor business rules stacked in layers, and pick their way through environments crowded with irrelevant and misleading records — the slice of cross-application orchestration that genuinely resembles production.
Grade the Final State, Not the Answer
The methodology page pulls no punches: “No LLM-as-judge. No vibes.” Every point comes from fixed assertions run against the final environment state — in the page’s words, “The agent’s reply isn’t scored — the data is.” Two tools are all agents receive: a BM25 keyword search over API schemas (the top 5 candidate endpoints come back) and an execute tool styled after curl/fetch. Behind the simulated APIs, Pydantic models act as the source of truth, keeping runs reproducible.
task_completed_correctly is the official metric and holds only when every assertion passes; partial_credit exists for diagnostics alone while doubling as an RL training signal. The assertion set mixes in negative checks, so shotgun reward hacking — emailing everyone rather than the specified recipients — gets shut down. Between runs, variance stays near 1%.
Leaderboard: GPT 6 Astra Out in Front, the Rest Trailing Low
Key entries from the leaderboard’s top ten (version 1.0.6, 95 model configurations in all):
- GPT 6 Astra (Max) tops the table at 41.4%, costing $1.77 per task
- The same model in its other settings: (XHigh) 38.96%, (High) 37.14%, (Medium) 34.09%
- Claude Fable 5.1 (Max), with Opus 5 as the fallback: 31.4% at $2.45 per task, and Opus 5 takes over about 40% of tasks (260 of 657)
- Gemini 3.7 Flash (High) reaches 30.44% at $0.61 per task; Gemini 3.8 Flash posts 29.68% across two settings
- GPT-5.6 Sol (Max) sits at 28.77%
Split by domain, Operations marks the high point at 62.0% (GPT 6 Astra Max), with Marketing next at 50.0%, then Finance at 43.33%, Sales at 40.2%, and Support at 39.0%. HR is the one domain an OpenAI model does not lead — that goes to Gemini 3.8 Flash (Medium) at 26.67%. On price-performance, the Gemini Flash family holds between $0.55–0.62 per task, roughly a third of Astra (Max)‘s spend.
False Confidence Tops the Failure List
The finding that stands out most is the pattern the authors name false confidence: a model announces success while the task has in fact failed. It makes up 72% of Opus’s failures, 91% of Gemini’s, and 84% of GPT-5.4’s — the agent says the work is done while the world state says otherwise. The recurring list goes on: quitting after the first search comes back empty, placing data wherever intuition says it “should” (CRM versus Sheets), stopping partway through a list yet summarizing it as finished, and restating instructions rather than executing them as written.
The benchmark compresses it into one line: “Like production, mostly-right is still wrong.”
Three Lessons for Agent Builders
The takeaways come in threes. To start, end-state verification should sit inside your eval pipeline — conversational quality and task completion can come apart entirely, and this dataset is the proof. Next, give cost per task first-class status: within the same ~30% score band, the Flash family spends a third of what Astra (Max) does, and that gap is real headroom for model routing. Finally, a model upgrade on its own will not cure false confidence — putting verification steps at the tool layer and requiring agents to re-read state usually costs less than buying a bigger model. In our 2026 opening outlook, we predicted agent deployments would run into a last-mile verification wall first; AutomationBench has since attached hard numbers to that wall.