Summary: AWS’s September 11, 2026 benchmark compares OpenAI models on Amazon Bedrock and OpenAI API deployments by the observed cost of a correct or passing result, rather than token price alone. Across AIME, agent web research, and a 48-task GDPval slice, Luna’s Bedrock deployment was especially economical after price cuts, while turn count and retries often dominated agent costs. The authors caution that samples were small and configurations were practical deployments, including reasoning off for Bedrock models.
Dollars per million tokens is how nearly every team sizes up a model, and the AWS Machine Learning Blog says that is the wrong unit to compare. A production workload never buys tokens; it buys an outcome — the ticket that gets resolved, the research brief that gets finished, the summary a reviewer will sign. Between the price list and that outcome sit three multipliers it never prints: how often the model is correct, how many tokens it spends getting there, and how many turns the job takes an agent. A September 11, 2026 post lays out an open-source harness measuring all three.
What the harness puts a number on
Five models, one code path. The benchmarking harness drives the OpenAI Responses API against three Amazon Bedrock deployments — gpt-5.6-luna, gpt-5.6-terra, gpt-5.6-sol — plus two cost-optimized OpenAI API baselines, gpt-5.4-mini and gpt-5.4-nano. What that shows, the AWS authors say outright, is practical deployments compared rather than intrinsic model capability: reasoning was off for the Bedrock models, and the API baselines ran at their defaults.
The harness collects three kinds of evidence. Accuracy and cost for a single call, across AIME, GPQA Diamond, and MMLU-Pro. Agent trajectories over many turns on live web-research tasks. And professional deliverables graded against rubrics, from a GDPval slice of 48 tasks. Scoring pairs deterministic checks with an LLM judge, and every figure traces to a timestamped result file.
The unit that matters is cost per right answer
Every benchmark gets the same arithmetic: divide the money spent on all attempts, successful and failed, by how many answers came out right. Accuracy first, on AIME — sol reaches 75 percent where mini manages 37 percent, and stays ahead on GPQA Diamond (68 against 43 percent) and MMLU-Pro (82 against 59 percent).
More striking is what token efficiency did before any price changed. Even at its original list price, luna’s cost per correct AIME answer already sat 25 percent below mini’s here, since reasoning off meant fewer billed tokens. Then came the July 30, 2026 price reduction for GPT-5.6 Luna and Terra on Amazon Bedrock — 80 percent off luna, 20 percent off terra — after which luna’s recorded cost per correct AIME answer reads $0.0021 against mini’s $0.0139. Luna also beat nano there, despite nano’s marginally lower nominal token price.
Sample sizes run 48–198 items, the authors warn, so read small gaps as direction, not a verdict.
Why turn count moves the bill
Single-call arithmetic collapses once an agent is in the loop. History is client-managed here, under store: false, so every turn carries the system prompt, earlier tool results and the accumulated conversation back over the wire. Context per turn rises roughly linearly, and billed input piles up close to quadratically with turn count.
Take the 50-question DeepSearchQA sample, run with real web_search and fetch_page tools. Mini burned the most turns per question — 7.6 of them, largely re-search loops — and finished at 2.3× terra’s input-token volume, 114k tokens against 50k. Terra’s pricier tokens were repaid by fewer turns and better answers: $0.31 per passing answer against mini’s $0.40, with mean F1 of 0.50 against 0.39. Luna landed at $0.05 per passing answer, against mini’s $0.40. Nano’s cheaper tokens bought it nothing: an 18 percent pass rate left its observed cost per passing answer at $0.07.
Whenever agents chain tool calls — research passes, multi-hop lookups, retrieval that iterates — a whole trajectory’s cost belongs in the same spreadsheet as one call’s. We made the same argument about prefix-aware routing on SageMaker: what your traffic looks like decides the bill, not the sticker price.
When paying more still makes sense
Of everything measured, GDPval comes nearest to work that ships: real occupational deliverables, scored against human-written rubrics, with 70 percent of weighted points needed to pass. With reasoning off, all three gpt-5.6 configurations beat mini and nano, the widest margins in law, nursing, and financial advice. Luna cleared 27 of the 48 deliverables, mini 20; per passing deliverable the cost was $0.010 for luna, $0.030 for mini, $0.012 for nano.
Terra and sol score higher on rubrics, and charge more for each pass. Whether that extra spend earns its keep turns on what review and rework cost you — a figure only your team holds.
What to do about it
Run the harness against your own tasks before deciding anything — the AWS authors’ own advice, and the fair conclusion to draw. Their samples are small, the latency figures capture one region at one moment, and an 8,192-token output cap left several GDPval deliverables truncated. Loosening it could raise quality and expense together, so both deserve a test.
Teams running mini or nano have a better first move than migrating: instrument what a successful outcome costs on the workload you actually serve, counting turns. Which model wins is not something a pricing page can answer.