Summary: OpenAI published a coding-evaluation audit on July 8, 2026 and withdrew its recommendation of SWE-Bench Pro, estimating that about 30 percent of the benchmark’s tasks are broken. Automated review, agent-assisted investigation, and human annotation traced failures to overly strict tests, underspecified or misleading prompts, and low-coverage tests, with public-split pass rates rising from 23.3 percent to 80.3 percent in eight months. OpenAI recommends carefully treating existing results, building benchmarks expressly for model capability testing, and using capable models to audit prompts, tests, patches, and traces.
On July 8, 2026, OpenAI put out “Separating Signal from Noise in Coding Evaluations” and did what large labs almost never do: it publicly disowned its own earlier stance. The bottom line is unambiguous — “we estimate that ~30% of SWE-bench Pro tasks are broken” — and the recommendation to adopt the benchmark was formally withdrawn. The awkward part is that SWE-Bench Pro was the benchmark OpenAI itself had put forward to succeed SWE-bench Verified for coding evaluation.
A second figure reads just as badly. In eight months, frontier models’ pass rate on the benchmark’s public split jumped from 23.3% to 80.3%. Inflation at that pace is seldom just better models: either the tasks are leaking, or the grading accepts partial work — both indict the benchmark.
A Public Walk-Back You Rarely See
Size up the audit: an automated pipeline sent 286 suspect tasks into deep review and ended up judging 200 of 731 (27.4%) broken; human annotators drew the line further out at 249 (34.1%). Five trained engineers examined every task independently, and on the deeply checked ones, “not broken” never held majority status. Weighed against the premise of OpenAI’s own Preparedness Framework — that capability evaluations have to be trustworthy — the post admits the exam the company relied on was itself unsound. Retractions being rare is what makes this one matter: labs normally rebut criticism of a benchmark rather than pull it.
How the Tasks Went Bad
The trouble begins with provenance. SWE-Bench Pro mines its tasks from real open-source pull requests — PRs “originally created for human collaboration,” in which maintainers and contributors exchange dozens of comments before anything merges. Description, merged code, and unit tests therefore often fail to compose a tidy, self-contained evaluation task. A PR test is there to “validate a specific change,” not to lay down an acceptance standard independent of implementation; repurpose that conversational byproduct as a machine exam and the noise arrives with the material. The defect is structural: scraped benchmarks inherit every assumption the original conversation rested on, and none of those were ever written down.
The Four Ways Tasks Break
Defective tasks fall into four kinds. Overly strict tests demand implementation details the prompt never states, rejecting functionally correct solutions. Underspecified prompts require what no reasonable reading of the task implies. Low-coverage tests check too little and let incomplete fixes through. Misleading prompts steer models toward behavior the tests contradict. One brutal illustration from the post: an OpenLibrary task whose example showed one leading space while the hidden tests demanded two, so a submission mirroring the example exactly failed. Each mode bends the signal its own way — strictness penalizes correct answers while leniency rewards incorrect ones — and a benchmark mixing both hides each.
Auditing in Three Layers
The method stacks three layers. An automated filter first reviews prompts, model attempts, and grading tests side by side to surface suspect tasks. Next comes human-supervised agent review: Codex-based investigator agents, holding repository and environment access, actually execute the tests and study failed attempts across repeated passes before a researcher decides. Finally, a human annotation campaign assigns five trained, experienced software engineers to each task to judge from the problem statement, the tests, and the gold patch, escalating disagreements. Human and agent category judgments coincided 74% of the time; on low-coverage tests humans flagged 9.4% of the benchmark against the agents’ 4.1% — experienced eyes still catch blind spots machine audits miss.
Takeaways for the Evaluation Ecosystem
Three recommendations deserve copying. First, read SWE-Bench Pro results with care — the endorsement is withdrawn. Second, write new benchmarks “by experienced software developers specifically to test model capabilities,” with difficulty and realism kept under human control end to end. Third, put increasingly capable models to work auditing prompts, tests, patches, and traces at scale, because spotting flaws keeps getting dramatically cheaper. The bar for a good evaluation: “hard to game, easy to trust, and genuinely reflective of model capability.” For teams that shop by leaderboard scores, the lesson is plain — the exam itself has become infrastructure needing upkeep. There is a self-interested reading too: OpenAI builds the models that do the auditing, and discrediting the incumbent exam raises the value of its own tooling. Either way, better data wins.