Key takeaways
- Eand study in the renowned magazine Nature shows that ChatGPT Health does not provide appropriate emergency advice in more than half of emergency cases.
- The AI tool performs well on textbook examples such as strokes, but fails when symptoms become less clear-cut.
- When reports of suicidal behavior occur, the system responds inconsistently, meaning crucial helplines do not always appear.
ChatGPT Health fails to refer patients to the correct acute care in more than half of emergency cases. A new study from Mount Sinai in New York shows that OpenAI’s AI tool underestimates the urgency, especially for less clear symptoms. As a result, millions of users who link their medical records may receive life-threatening advice to wait.
Blind spots in acute emergencies
Research leader Ashwin Ramaswamy states that the language model mainly struggles with situations in which the danger is not immediately visible. During tests with sixty clinical scenarios, the AI recognized the early signs of respiratory failure in an asthma patient, but still advised to wait rather than seek help. This contrast is striking, because the chatbot responds correctly in obvious emergencies such as a severe allergic reaction. The researchers conclude that the technology is currently not a substitute for a doctor’s clinical judgment.
No total ban
The study also revealed a disturbing pattern in the detection of self-harm. ChatGPT Health should immediately refer users with suicidal thoughts to helplines. In practice, however, the banner with help information appeared irregularly. Paradoxically, the system responded more reliably to general reports than to patients who already mentioned a specific method of self-harm. Especially when the severity of the situation increased, the built-in safety mechanisms were more often omitted.
Despite these results, the authors do not advocate a total ban on AI health tools. Co-author Alvira Tyagi points out that these technologies are already in the hands of millions of people. According to her, it is essential to learn how to thoughtfully integrate these resources into healthcare without seeing them as substitutes for human expertise. Because AI models are continuously evolving, constant evaluation remains necessary. For the time being, the advice is to contact a healthcare provider immediately in case of alarm symptoms such as chest pain or shortness of breath and not to rely on a chatbot.