When faced with symptoms like a runny nose and cough, some may think it’s a cold. A visit to the doctor isn’t necessary, so they turn to Google and WebMD for further reassurance. With advances in AI, some may now be tempted to switch from “Dr. Google” to “Dr. ChatGPT,” but can OpenAI’s AI-powered chatbot provide accurate medical advice?

Researchers at Western University wanted to answer this question and investigate whether ChatGPT could become a reliable resource in healthcare and medical education. The study, led by Dr. Amrit Kirpalani, professor at the Schulich School of Medicine & Dentistry, was recently published in PLOS One and found that ChatGPT was only 49% accurate in providing the correct diagnosis.

While they concluded that ChatGPT cannot yet be used as a reliable medical diagnostic tool for complicated cases, their study showed that it is capable of summarizing complex medical topics in an easy-to-understand manner, which could be beneficial for educators and medical staff to provide medical information in an easy-to-understand format.

“For me, the most important finding is that ChatGPT provided its answers in a very simple and easy-to-understand way,” Kirpalani said. “I think that’s important because you can see that it can be used as a great tool to help people learn and understand medical cases.”

As part of the study, ChatGPT was asked to diagnose 150 cases as part of the Medscape Clinical Challenges, which are tests designed to test the diagnostic skills of healthcare professionals. Medscape is a public platform with many complex cases where doctors vote on what they think the right answer is. The research team, which included third-year medical students Ali Hadi, Edward Tran and Branavan Nagarajan, created prompts that asked ChatGPT to select the correct diagnosis in a multiple-choice format and provide a rationale.

The chatbot was fed information such as patients’ medical history, physical exam results, and laboratory or imaging tests. Researchers found that it had difficulty interpreting test results and sometimes missed important information relevant to diagnosis. However, the chatbot was helpful by offering next diagnostic steps and making medical information more accessible.

More research needed to “use AI responsibly”

It’s clear from the study that more research and advancements are needed before AI can be used as another tool in medical diagnosis. And as new AI models advance and improve, Kirpalani stresses the importance of AI skills.

“AI literacy is important for patients, providers, educators and students because we need to understand how to use AI responsibly and how it can be applied and used for healthcare and medical education purposes,” says Kirpalani.

Regardless of the accuracy of these online resources, Kirpalani stressed the need to evaluate responses from the Internet and verify them against reliable, peer-reviewed sources.

“I would say we may already be at the point where we need guidance on prompt engineering – where an instruction is structured that can be interpreted and understood by a generative AI model,” Kirpalani said.

“We will need comprehensive oversight of their use to ensure patient safety and to ensure that (this type of AI technology) is introduced wisely.”