New research reveals that AI explanations can support medical professionals while potentially misleading users with less expertise in healthcare diagnosis.
AI-generated explanations can encourage non-experts to accept incorrect medical recommendations, according to research involving MIT and several partner institutions.
Published in Nature Medicine, the study examined how different forms of explainable AI affected 623 members of the public and 153 primary care physicians performing skin-disease diagnosis tasks.
Participants received assistance in several formats, including a model prediction with a confidence score, similar medical images, heat maps highlighting relevant areas and explanations generated by a large language model.
AI assistance generally improved diagnostic accuracy and reduced performance differences across skin tones. However, the benefits varied significantly according to users’ medical expertise and the accuracy of the model’s recommendation.
Non-experts tended to follow AI-generated diagnoses even when they were wrong. They also found vague or generic language-model explanations more convincing, suggesting that a plausible explanation can strengthen trust without helping users assess whether a recommendation is correct.
Primary care physicians were more resilient when the AI made mistakes and were able to apply their own clinical knowledge. They achieved their strongest results when shown the model’s prediction without an accompanying explanation.
The order in which information was presented also affected decisions. Showing the AI recommendation before asking users to make their own assessment increased anchoring, producing worse results when the model was incorrect for both groups.
Why does it matter?
The findings show that making medical AI more explainable does not automatically make it safer. Explanations can increase automation bias among less experienced users, meaning healthcare systems must adapt AI interfaces to different levels of expertise and preserve independent human judgement.
Would you like to learn more about AI, tech, and digital diplomacy? If so, ask our chatbot!
