Factual accuracy versus the health literacy barrier: a blinded neurotological assessment of ChatGPT in vertigo education

Abstract

Background: Patients increasingly utilize artificial intelligence (AI) models like ChatGPT to seek medical information regarding complex symptoms such as vertigo. The reliability of these AI platforms in providing accurate, comprehensible, and safe clinical information remains a subject of ongoing investigation. Objectives were to evaluate the clinical accuracy, comprehensiveness, and readability of ChatGPT’s responses to common patient inquiries regarding vertigo and vestibular disorders.

Methods: Fifty frequently asked questions (FAQs) concerning vertigo were curated from high-traffic patient forums, clinical encounters, and search engine trends. These questions were categorized into four domains: Diagnosis, treatment, home exercises/lifestyle, and prognosis. The FAQs were inputted into ChatGPT-4. Responses were independently evaluated using a 5-point Likert scale for accuracy and comprehensiveness. Readability was assessed using the Flesch-Kincaid grade level (FKGL) and Flesch reading ease (FRE) scores.

Results: ChatGPT demonstrated high overall accuracy (mean score 4.38±0.62) and comprehensiveness (4.21±0.74). The model performed best in the home exercises/lifestyle category (accuracy 4.65±0.45) and scored lowest in the diagnosis domain (accuracy 4.05±0.82), where it occasionally failed to distinguish atypical features of central vs. peripheral vertigo. The mean FKGL was 12.4±1.8, and the mean FRE was 41.2±10.5, indicating a reading level appropriate for college students but potentially difficult for the general public.

Conclusions: ChatGPT-4 generates highly accurate and comprehensive answers to common patient questions about vertigo, making it a valuable supplementary educational tool. However, its sophisticated vocabulary and occasional diagnostic ambiguity emphasize the necessity for physician oversight.