Study finds AI reasoning models make over 34% more false-positive diagnoses than matched standard models

Nature Communications · Research paper

Read the source (opens in a new tab)
What happened
Researchers from the University of Hong Kong, Tsinghua University, Harvard Medical School and others published a Nature Communications study on October 29, 2026 comparing reasoning and standard AI models in clinical diagnosis.
Key points
  • On a benchmark built from real-world electronic health records, six matched model pairs across 12 specialties were compared.
  • Reasoning models made more accurate primary diagnoses but over 34% more false positives and about 40% lower reproducibility; the authors say the trade-off held across prompt designs and settings.
  • In a 15-physician study, letting the reasoning model widen the diagnosis while the clinician checked it raised accuracy by 20% and cut false positives by 44%.
  • The authors say pairing reasoning models with physician oversight needs "prospective validation before clinical deployment."
Evidence
Peer-reviewed, open-access paper. The figures are from its abstract; the full results and the specific models tested were not reviewed for this entry.
Uncertain
The authors say the results were "obtained under controlled retrospective conditions," not in live patient care.

See it in the latest record