Fourth-year ophthalmology resident at Emory University School of Medicine found something surprising when looking into the accuracy of online medical advice.
NPR reported that the resident physician and two colleagues evaluated the accuracy of AI chatbot ChatGPT in diagnosing eye-related complaints. The docs wondered if AI could help make assessing patients easier, and they'd all experienced patients seeking out "Dr. Google," according to NPR.
Lyons told NPR that "any number of terrible things could be going on based on the symptoms that they're experiencing."
In June, Lyons and company reported in medRxiv, online health science preprint publisher, that ChatGPT outperformed online "symptom-checkers" like WebMD, and compared pretty well to human doctors in triaging patients' complaints.
In spite of the much-publicized "hallucination" problem with AIs — that AIs, when they don't know the answer, will just make up a statement — the Emory study found the most recent version of ChatGPT made no "grossly inaccurate" statements, when presented with a standard series of complaints — the kind the doctors themselves are trained with.
The relative proficiency of ChatGPT, introduced to the world back in November 2022, was a surprise to Lyons and his co-authors.
ChatGPT "is definitely an improvement over just putting something into a Google search bar and seeing what you find," said co-author Nieraj Jain, assistant professor at the Emory Eye Center who specializes in vitreoretinal surgery and disease.
But the study's findings underscore a challenge facing healthcare as it looks into the pros and cons of generative AI, the kind used by ChatGPT.
The accuracy may be an improvement over Dr. Google and WebMD, but there are still question about integrating the new tech into the wild world of patient care with guardrails to keep patients safe.