Summary: A Harvard Medical School and Beth Israel Deaconess study published by Science on April 30, 2026, found that OpenAI’s o1 outperformed two internal-medicine attendings on ER diagnosis tasks built from 76 real patients’ electronic records under blinded scoring. o1 produced exact or near-exact triage diagnoses in 67% of cases versus 55% and 50% for the physicians; with fuller records the 82%-versus-70%-to-79% gap was not statistically significant. The text-only study’s authors said it does not support live ER decision-making, and critics noted the physicians were not emergency specialists.
Science carried new work by a Harvard Medical School and Beth Israel Deaconess Medical Center team on April 30, 2026, and the headline finding does not mince words: diagnosing emergency-room cases, OpenAI’s o1 reasoning model beat two attending physicians. TechCrunch covered it on May 3, and the general audience followed. This was no leaderboard stunt: the inputs were electronic health records from real ER arrivals, and scorers never knew whether a diagnosis came from a human or a model. Teams still deciding where clinical AI belongs should weigh the numbers and the limits equally.
The Setup Behind the Experiment
Arjun Manrai, who heads an AI lab at Harvard Medical School, and Adam Rodman, a physician at Beth Israel, led the work. The team picked 76 patients who arrived at the Beth Israel emergency room and asked o1, GPT-4o, and two internal medicine attendings to diagnose them at identical checkpoints, working from the electronic record as it stood at each moment. Two more attendings graded every diagnosis without ever learning its origin. No input was pre-processed: the models saw exactly what clinicians saw. The authors stake out a bold position: large language models “have eclipsed most benchmarks of clinical reasoning.”
Reading the Results
Triage — the phase with the least to go on — is where o1 opens its lead: an exact or near-exact diagnosis in 67% of cases, against 55% and 50% for the two physicians. With more clinical detail filled in, o1 moved up to 82% while the physicians sat between 70% and 79%, a difference the study marks as statistically insignificant. The treatment-planning substudy is starker: across 5 case studies and 46 doctors, the AI scored 89%, while physicians limited to conventional resources — search engines counting among them — managed only 34%. One case says it best: a lung-clot patient’s symptoms kept worsening, and o1 correctly connected them to a lupus history the human doctors had passed over.
Why Triage Is the Model’s Strongest Moment
Nothing in the ER carries less information or more urgency than triage: the patient has barely arrived, labs are still out, and the clock is unforgiving. A model does comparatively well here: it weighs every thread of the record at once, and a rare but decisive line of history never slips away to fatigue or an anchored early guess. Speaking to The Guardian, Rodman placed large language models among “the most impactful technologies in decades” and foresaw a coming “triadic care model — the doctor, the patient, and an artificial intelligence system.” Manrai chose caution: “I don’t think our findings mean that AI replaces doctors” — he sees instead “a really profound change in technology that will reshape medicine.”
The Pushback, and Where the Study Stops
The hardest critique came from ER physician Panthagani: “an interesting AI study that has led to some very overhyped headlines.” Her objection goes to the baselines: internal medicine attendings are not emergency physicians, and someone working an ER is not there to name the ultimate diagnosis — the priority is ruling out what could kill the patient fast, a different role by nature. There is a technical ceiling too: the models consumed text only, blind to the patient’s appearance, distress, and bedside cues — by the authors’ own description, a second opinion based on paperwork. The paper states plainly that its findings do not yet support live ER decision-making and calls for prospective real-world trials. Rodman also flagged a governance hole: there is “no formal framework right now for accountability” around AI diagnoses, and on life-or-death calls patients still expect a human in the decision.
Takeaways for Clinical and Product Teams
Adoption trends deserve as much attention as the study itself: close to one in five US physicians already leans on AI to help diagnose, and in the UK 16% of doctors use it every day, another 15% weekly. Clinical reasoning runs on dense documents and fragmented information — terrain where consumer-grade models now settle in comfortably. Three things for builders. Triage and differential diagnosis are the early touchpoints with the highest value, because what goes in is nearly all text. Blinded scoring plus the “exact or near-exact” rubric is a pragmatic setup that transfers into an in-house evaluation flow. Accountability and specialty matching demand answers before deployment — the controversy over internal medicine attendings as the baseline becomes a textbook study-design lesson for the next iteration. The paper never claims AI can run an ER today; its center of gravity is that AI has become good enough to justify formal clinical testing.