STAT+: Clinical chatbots are taking medicine by storm. Should doctors trust them?

STAT PlusHealth tech Clinical chatbots are taking medicine by storm. Should doctors trust them? What benchmarks do — and don’t — say about clinical LLMs from OpenEvidence, Doximity, and UpToDate Manage alerts for this article Email this article Share this article By Katie PalmerJuly 29, 2026 Health Tech Correspondent Katie Palmer[email protected]Katie covers the impact of health technology on patients, clinicians, and businesses.

Her stories explore the price tag of clinical AI, digital health at the FDA, and the boom in direct-to-consumer telehealth. Confidential tips can be sent on Signal at palmer.01. Hundreds of thousands of U.S.

doctors use clinical large language models, pitched by companies like OpenEvidence, Doximity, and UpToDate as an antidote to the dangers of hallucination-prone generalist models from Big Tech. Yet few studies have pitted them against each other — and this summer brought a high-profile head-to-head. Researchers from NYU Langone Health had tested general and clinical models, including OpenEvidence and UpToDate Expert AI, on three sets of clinical questions.

The findings, published in Nature Medicine in June: The clinical AI performed worse than the general models. Advertisement The results rang out like a gunshot. “I’ve never seen a single paper trigger the kind of reactions this one has in the health AI community,” wrote Kaiser Permanente’s vice president of AI and emerging technologies on LinkedIn.

The paper’s findings, like all science, are subject to interpretation and debate — but many online reactions treated them more like a clear victory for general frontier models. STAT+ Exclusive Story Already have an account? Log in This article is exclusive to STAT+ subscribers Unlock this article — and get additional analysis of the technologies disrupting health care — by subscribing to STAT+.

Already have an account? Log in Monthly $39 Totals $468 per year $39/month Get Started Totals $468 per year Starter $30 for 3 months, then $399/year $30 for 3 months Get Started Then $399/year Annual $399 Save 15% $399/year Get Started Save 15% 11+ Users Custom Savings start at 25%! Request A Quote Request A Quote Savings start at 25%!

2-10 Users $300 Annually per user $300/year Get Started $300 Annually per user View All Plans To read the rest of this story subscribe to STAT+. Subscribe Log In Artificial intelligence, health tech, STAT+ Submit a correction requestReprints Katie Palmer Health Tech Correspondent Katie covers the impact of health technology on patients, clinicians, and businesses. Her stories explore the price tag of clinical AI, digital health at the FDA, and the boom in direct-to-consumer telehealth.

Confidential tips can be sent on Signal at palmer.01. STAT+ Newsletter A hitchhiker's guide to the health care AI galaxy Recommended Biotech July 29, 2026 STAT Plus: In the biotech industry, a most-wanted fugitive was long hiding in plain sight First Opinion July 29, 2026 The VA could help America win the biotechnology race with China Advertisement First Opinion July 29, 2026 Prolonged hospital stays can leave children isolated. It needs to change Health July 28, 2026 STAT Plus: American Diabetes Association, facing calls for resignations, asks for patience Politics July 28, 2026 Six key issues we’re watching at Fauci’s hearing Subscriber Picks