Brain Sciences, Vol. 15, Pages 392: Assessing the Accuracy of ChatGPT in Answering Questions About Prolonged Disorders of Consciousness

Brain Sciences doi: 10.3390/brainsci15040392

Authors:
Sergio Bagnato
Cristina Boccagni
Jacopo Bonavita

Objectives: Prolonged disorders of consciousness (DoC) present complex diagnostic and therapeutic challenges. This study aimed to evaluate the accuracy of two ChatGPT models (ChatGPT 4o and ChatGPT o1) in answering questions about prolonged DoC, framed as if they were posed by a patient’s relative. Secondary objectives included comparing performance across languages (English vs. Italian) and assessing whether responses conveyed an empathetic tone. Methods: Fifty-seven open-ended questions reflecting common caregiver concerns were generated in both English and Italian, each categorized into one of three domains: clinical data, instrumental diagnostics, or therapy. Each question contained a background context followed by a specific query and was submitted once to both models. Two reviewers evaluated the responses on a four-point scale, ranging from “incorrect and potentially misleading” to “correct and complete”. Discrepancies were resolved by a third reviewer. Accuracy, language differences, empathy, and recommendation to consult a healthcare professional were analyzed using absolute frequencies, percentages, the Mann–Whitney U test, and Chi-squared tests. Results: A total of 228 responses were analyzed. Both models provided predominantly correct answers (80.7–96.8%), with English responses achieving higher accuracy only for ChatGPT 4o on clinical data. ChatGPT 4o exhibited greater empathy in its responses, whereas ChatGPT o1 more frequently recommended consulting a healthcare professional in Italian. Conclusions: Both ChatGPT models demonstrated high accuracy in addressing prolonged DoC queries, highlighting their potential usefulness for caregiver support. However, occasional inaccuracies emphasize the importance of verifying chatbot-generated information with professional medical advice.

Source link

Sergio Bagnato www.mdpi.com