Evaluating the performance of ChatGPT in responding to patient eye health queries
Theme: Glaucoma
What: Glaucoma
Part of: Glaucoma IV: New Tech / Glaucome IV: Nouvelles technologies
When: 6/1/2024, 11:15 AM - 12:45 PM
Where: Room | Salle 801
Abstract
Purpose: We evaluated the ability of ChatGPT, an AI Chatbot, to respond to patient eye health queries.
Study Design: Cross-sectional study.
Methods: We conducted an analysis of eye health questions and physician responses posted on the American Academy of Ophthalmology (AAO) ‘Ask an Ophthalmologist’ forum from January 2015 to December 2022. We compared board-certified ophthalmologists’ responses to ChatGPT (version GPT-3.5, OpenAI) responses. Primary outcomes included automated AI-rated similarity and accuracy of ChatGPT responses, relative to ophthalmologist responses. Secondary outcomes included readability, empathy, and length of ChatGPT responses compared to ophthalmologist responses.
Results: A total of 1,079 questions and responses from 41 board-certified ophthalmologists were assessed. The mean similarity score for ChatGPT responses compared to the AAO responses was 68% (SD=22%). The mean accuracy of ChatGPT responses, when AAO responses were considered the ‘gold standard’, was 90% (SD=8%). Ophthalmologist’s responses had a lower mean Flesch-Kincaid Grade Level (Grade 11.0 [SD=2.7] v Grade 13.6 [SD=2.0], t=27.7, p<0.001) than chatbot responses, making them easier to understand. Ophthalmologist responses were significantly shorter than ChatGPT responses (80.6 [SD=56.4] words v (130.1 [SD=48.2] words, t=21.9, p<0.001). Empathy scores of ChatGPT responses were rated as not being significantly different from ophthalmologist responses (2.46 [SD=0.50] v 2.43 [SD=0.48], t = 0.359, P =0.72).
Conclusions: Our findings suggest that ChatGPT has acceptable similarity and good accuracy compared to ophthalmologists’ responses for answering patient eye health queries. AI chatbots may be useful in drafting initial responses to patient ocular concerns, potentially increasing efficiency and reducing workload.
Presenter(s)
Presenting Author: Mostafa Bondok
Additional Author(s):
Mostafa Bondok, Faculty of Medicine, University of British Columbia,
Rishika Selvakumar, School of Population and Public Health, University of British Columbia
Nupura K. Bakshi, Department of Ophthalmology and Visual Sciences, University of Toronto; Department of Ophthalmology, St. Michael’s Hospital; Department of Ophthalmology, Mount Sinai Hospital
Tina Felfeli, Department of Ophthalmology and Visual Sciences, University of Toronto; The Institute of Health Policy, Management and Evaluation, University of Toronto
Evaluating the performance of ChatGPT in responding to patient eye health queries
Category
Glaucoma
Description
Presentation Time: 12:03 PM to 12:10 PM
Room: Room | Salle 801