Summary: A Reflex benchmark compared the same Claude model completing an admin task through browser screenshots and through structured APIs. The vision path averaged 550,976 input tokens and about 17 minutes, while the API path used 12,151 tokens and finished in 19.7 seconds. The article argues that interface design, not model choice alone, drives agent cost and reliability.
May 2026 brought a bucket of cold water for the computer-use crowd: Reflex, the Python full-stack framework, ran one benchmark in which a single Claude model handled a single admin-panel job two different ways — once as a vision agent clicking around a browser, once calling structured APIs. On input tokens the two approaches landed roughly 45x apart: clicking through the UI averaged 550,976 input tokens and about 17 minutes of work, while the API path wrapped up with 12,151 tokens in 19.7 seconds. Reflex open-sourced the benchmark code together with the write-up, and the piece hit the Hacker News front page on May 5, pulling in plenty of discussion.
Same Job, Different Interface
Everything except one variable was pinned down. The task ports react-admin’s Posters Galore demo: over a frozen dataset of 900 customers, 600 orders, and 324 reviews, the agent has to locate the customer named Smith who holds the most orders, approve every pending review of theirs, and flag their newest order as delivered. The interface is the only place the two paths diverge. On the vision side, Claude Sonnet drives browser-use 0.12 in vision mode, working from screenshots and clicks. On the API side it is the same Sonnet, except that a Reflex 0.9 plugin generates HTTP endpoints straight from the app’s event handlers, letting the model use tool calling against typed tools such as list_customers and update_order. Identical model, identical data, identical task — the interface is the single thing that changes.
How the 45x Piles Up
Put side by side, the numbers are ugly. The vision agent needed 53 steps on average (give or take 13), and each one ships a full-page screenshot back into the model, piling up 551k input tokens. The API agent makes a fixed 8 calls, spends 12,151 input tokens, and barely wobbles (give or take 27 tokens). Wall clock runs parallel: 1,003 seconds against 19.7. Changing models moves things only marginally: on the API path, Haiku completes the job in 7.7 seconds while staying under 10k input tokens. The catch is that a stronger model trims the price of each step, while the interface dictates how many steps exist — and every screenshot carries a fixed toll of thousands of tokens.
Two Bills the Vision Path Hides
Past the headline figures, the benchmark turned up two costs no invoice will ever list. To start, fed nothing but the plain prompt, the vision agent did not actually succeed: it caught just 1 of the 4 pending reviews, because the other three lived behind table pagination the model never scrolled through. Only after the author appended a 14-step written walkthrough of the UI did it pass all three runs — and that hand-written documentation is precisely the invisible engineering labor the vision route keeps demanding. On top of that, the vision path is wildly non-deterministic: its three runs spread from 407k to 751k tokens and from 749 to 1,257 seconds, which makes any single run meaningless as evidence. The authors are equally frank about the limits: 3 to 5 runs per sample, a single task, and although the endpoints came out of automation, the tool descriptions handed to the API agent were still authored by people.
Where Computer Use Keeps Its Place
Nothing here condemns vision agents; what decides the outcome is whether the app is yours. For internal tools you wrote yourself, structured APIs take it decisively — and frameworks such as Reflex that auto-generate the endpoint layer have removed the classic expense of hand-rolling an API or MCP layer. Whenever the target is a third-party SaaS product or a legacy system you cannot touch, screenshots and clicks stay the only entrance. Web data extraction runs into the same choice: a structured scraping API generally outperforms putting a browser agent behind the wheel, as covered in our Firecrawl scraping guide. For anyone driving real browsers on the open web, bot detection remains a permanent wildcard.
Three Lessons Developers Should Keep
First, shape interfaces around your agents rather than forcing agents to cope with interfaces built for humans: screenshots cost the same every time, step counts follow from the interface, and that is the true wellspring of the 45x. Second, judge computer-use cost by total tokens and their variance instead of the per-step rate; both the benchmark’s code and its raw results sit on GitHub, and the directional finding holds up. Third, a Haiku-class model behind a clean API interface outperforms a flagship model behind a vision interface — dollars aimed at interface engineering return more than dollars aimed at a bigger model.