On July 8, 2026, OpenAI published “Separating Signal from Noise in Coding Evaluations” and did something large labs rarely do: publicly overruled itself. The conclusion is blunt — “we estimate that ~30% of SWE-bench Pro tasks are broken” — and the company formally retracted its earlier recommendation to adopt the benchmark. The irony is hard to miss: SWE-Bench Pro was OpenAI’s own pick, previously recommended as the successor to SWE-bench Verified for coding evaluation.
Another number stings just as much. Within eight months, frontier models’ pass rate on the benchmark’s public split climbed from 23.3% to 80.3%. Score inflation at that speed is rarely just models getting better. Either the tasks leak, or the grading forgives partial work; both point back at the benchmark.
A Rare Public Self-Correction
The scale of the audit: an automated pipeline flagged 286 suspect tasks for deeper review and ultimately judged 200 of 731 tasks (27.4%) broken; human annotators went further, flagging 249 tasks (34.1%) as defective. Each task was reviewed by five trained engineers, and on the tasks examined in depth, “not broken” never became the majority human label. Against the premise of OpenAI’s own Preparedness Framework — capability evaluations must be trustworthy — this amounts to admitting the exam the company once used was itself faulty. The move matters because retraction is rare: labs usually answer benchmark criticism with rebuttals, not withdrawals.
Why Tasks Rot
The root cause is provenance. SWE-Bench Pro tasks are scraped from real open-source pull requests, which were “originally created for human collaboration”: maintainers and contributors trade dozens of comments before anything merges. Descriptions, merged code, and unit tests often fail to form a clean, self-contained evaluation task. A PR test exists to “validate a specific change,” not to define an implementation-agnostic standard of acceptance. Treat the byproduct of human conversation as a machine exam, and the noise is baked in from the start. The mismatch is structural, not accidental — a benchmark built by scraping inherits every assumption the original conversation made, and none of those assumptions are written down.
Four Failure Modes
The audit sorts broken tasks into four categories. First, overly strict tests: the tests enforce implementation details the prompt never mentions, so functionally correct solutions get rejected. Second, underspecified prompts: hidden tests demand requirements that no reasonable reading of the task implies. Third, low-coverage tests: too little checking, so incomplete fixes pass anyway. Fourth, misleading prompts: the task steers models toward behavior the tests contradict. The post offers a brutal example — an OpenLibrary task whose example showed one leading space while the hidden tests required two, so a submission that followed the example exactly simply fails. Each mode corrupts the signal in a different direction — strictness punishes right answers, leniency rewards wrong ones — and a mixed benchmark hides both.
A Three-Layer Audit
The methodology runs in three layers. The first is an automated filter that reviews prompts, model attempts, and grading tests together to flag suspect tasks. The second is human-supervised agent review: Codex-based investigator agents, with repository and environment access, actually run the tests and inspect failed attempts in repeated passes, with a researcher making the final call. The third is a human annotation campaign: five trained, experienced software engineers per task, judging from the problem statement, the tests, and the gold patch, with escalation for disagreements. Human and agent category judgments overlapped 74% of the time; on low-coverage tests, humans flagged 9.4% of the benchmark against the agents’ 4.1% — machine audits still miss blind spots that experienced eyes catch.
What This Means for Evals
Three recommendations worth copying down. Treat SWE-Bench Pro results with caution; the adoption advice is retracted. Build new benchmarks “by experienced software developers specifically to test model capabilities,” keeping difficulty and realism under human oversight throughout. And use improving models to audit prompts, tests, patches, and traces at scale, because flaws are now dramatically cheaper to detect. The bar the post sets for a good evaluation: “hard to game, easy to trust, and genuinely reflective of model capability.” For any team buying models off leaderboard numbers, the message is plain — the exam itself is now infrastructure that needs maintenance. There is also a self-interested read: OpenAI builds the models that audit, and discrediting the incumbent exam makes its own tools more valuable. Both readings still favor better data.
Sources
AI-assisted summary compiled from the sources above, reviewed by a human before publishing.