New study shows ChatGPT Health failed to identify medical emergencies
Of clinical scenarios requiring the ER, over 50% were told to stay home
There’s been a lot of speculation and hype around the role of AI tools in medical care, with Dr. Oz going so far as to say AI avatars could meet healthcare needs in rural areas. In January, ChatGPT got into the health game and launched a new product called ChatGPT Health, “a dedicated experience in ChatGPT designed for health and wellness.”
This week, a new study was published evaluating how well this tool does at a critical task: telling people if they need to see a doctor. It failed, miserably.
The study published in Nature Medicine this week provided several clinical scenarios to the chatbot and asked it to triage them — should the person stay home, make a routine appointment within a few weeks, make an urgent appointment within 24-48 hours, or go to the ER now.
ChatGPT Health under-triaged half of emergencies
The results were concerning. While the AI tool performed reasonable well for patients requiring routine or urgent visits, it failed at the extremes.
For clinical cases where the appropriate response would have been to tell people to stay home, ChatGPT recommended a doctor’s appointment 64.8% of the time. While not dangerous, this may contribute to unnecessary healthcare utilization in an already overwhelmed health system, not to mention asking people to spend time and money they may not have.
But more concerning was the under-triage at the other extreme: of the clinical cases that required emergent evaluation in the ER, ChatGPT Health recommended they go to the ER in only half of cases.
Digging into which cases the AI tool was getting wrong was revealing — while it performed reasonably well with “textbook emergencies” like stroke and anaphylaxis, it did less well with emergencies that were still early in their clinical course such as the beginning of diabetic ketoacidosis (a life-threatening complication of diabetes) or the beginning of an asthma exacerbation. A good ER doctor recognizes when you are about to go down a hill of death, and stops it. ChatGPT, apparently, can only tell once you’re going full speed down the hill.
The study also revealed how extra, irrelevant information can distract ChatGPT. They tested a case of a patient with suicidal thoughts, where a suicidal crisis guardrail is supposed to fire, urging the person to seek help and providing information for the 988 Suicide and Crisis Lifeline. When just the relevant patient information was provided, the appropriate suicide crisis guardrail fired correctly 100% of the time. BUT, if the scenario also included that the suicidal patient had normal routine lab work, ChatGPT came up with a different answer: your labs are normal and don’t explain these thoughts, and the suicide crisis guardrail failed to fire.
This is quite bad. Over 230 million people across the world ask ChatGPT questions about health every week, according to their website. While it’s true that AI tools have done reasonably well on medical student exams, this study reveals what every fresh medical graduate knows: the ability to pass an exam does not translate to an ability to care for patients. Boards exam questions provide predictable, cookie cutter scenarios designed to teach patterns. But real, breathing humans don’t always follow the textbook.
For simple questions, chatbots can be amazing tools. But when it comes to deciding if you need medical care, I still recommend talking to an old-fashioned human doctor.
Kristen Panthagani, MD, PhD, is completing a combined emergency medicine residency and research fellowship focusing on health literacy and communication. In her free time, she is the creator of the medical blog You Can Know Things, available on Substack and youcanknowthings.com. You can also find her on Instagram and Threads. Views expressed belong to KP, not her employer.
Kristen, thank you for highlighting this study and walking through it so clearly. It is an important paper and deserves careful attention.
The under-triage of early DKA and asthma exacerbation is concerning, and the inconsistent activation of the suicide crisis safeguard is particularly troubling. Those findings should not be minimized.
At the same time, a few nuances are worth emphasizing. The model correctly identified textbook emergencies such as stroke and anaphylaxis, suggesting that the problem is not basic recognition but threshold calibration. The failures were concentrated in trajectory-dependent conditions, where urgency depends on recognizing clinical progression rather than dramatic presentation. That is one of the hardest judgment calls in medicine, even for humans.
The improvement in accuracy when objective physiologic data were included also suggests a calibration problem rather than a complete failure of medical reasoning. The system appears to recognize many of the right signals but does not consistently weight them toward escalation at clinical extremes.
The suicide guardrail findings may be the most consequential issue. A safety system should respond more strongly when risk is clearer, not less. When escalation is inconsistent relative to severity, trust and safety both suffer.
To me, the takeaway is not that AI has no role in medicine, but that consumer-facing triage systems require stronger calibration at the edges and more robust, reliable crisis safeguards before deployment at scale.
Thank you again for bringing serious analysis to a topic that is often discussed in extremes.
Thank you so very much for presenting this information. Hands on medicine cannot be replaced but yes can be supplemented with AI. The subtle nuances that a patient may present have a much higher chance of being detected by another human being with visual contact, the human touch, hearing what a patient sounds like when talking or breathing, a physical exam , the sense of smell & years of experience & gut intuition. We have 5 senses for so many reasons & AI does not.