Patients increasingly have a legal right to understand why an AI system recommended a particular diagnosis or treatment, but the technical reality of explaining these decisions remains largely undefined. The European Union's AI Act provides a legal foundation for transparency in high-risk medical systems, yet clinicians and hospitals struggle to deliver explanations that are both technically accurate and actually useful to patients. This growing gap between regulatory requirements and practical capability is creating a fundamental tension in healthcare AI adoption.

Why Can't Doctors Explain What AI Recommends?

The core problem lies in what researchers call the interpretability trade-off. The most accurate artificial intelligence models often operate through millions of parameters that are essentially impossible for humans to trace fully. When a deep learning system analyzes a medical image or patient data, it processes information through layers of mathematical operations that don't translate neatly into human-readable explanations. Forcing simpler, more explainable models could potentially sacrifice diagnostic accuracy, creating a direct conflict with patient safety .

Even experienced clinicians face a second barrier: automation bias. Research indicates that incorrect AI suggestions can pull doctors toward an incorrect diagnosis regardless of their experience level. When a clinician has already deferred to an algorithm's recommendation, any explanation they provide may not reflect an independent clinical assessment. This means the explanation a patient receives might not be grounded in the doctor's own reasoning .

Beyond the technical challenges, there's a fundamental communication problem. Between 22% and 58% of European Union citizens report difficulty understanding health information. Providing technical detail on algorithmic logic often leads to cognitive overload rather than informed consent. Patients may receive an explanation that is technically correct but practically useless for making a healthcare decision .

What Would Actually Help Patients Understand AI Decisions?

Experts argue for a paradigm shift away from checkbox compliance toward decision-relevant clarity. Rather than simply documenting that an explanation was provided, healthcare systems need to ensure patients can genuinely use the information to make choices. This requires several concrete changes:

• Co-design with Patients: Developers must test explanation systems with actual patients and patient advocates to ensure they meet real-world needs, not just regulatory requirements
• Institutional Time and Training: Health care systems need to allocate specific time for AI discussions and train staff to navigate these complex conversations with patients
• Comprehension Standards: Policy makers should prioritize digital health literacy and develop standards that measure whether a patient can actually use the information provided to make a choice

A truly useful patient-facing explanation must address three core questions: what the system recommends, how confident it is in that recommendation, and what the known performance gaps are for specific populations. For example, if an AI diagnostic tool performs differently for patients of different ages or racial backgrounds, patients deserve to know that variation exists .

"The EU AI Act provides the legal foundation, but the capacity to deliver an explanation that a patient can genuinely use is shaped by forces the law alone cannot govern. What patients need now are answers they can use," the report concludes.

Anshu Ankolekar, PhD., JMIR Correspondent

How to Build AI Healthcare Systems Patients Can Actually Trust

• Test Explanations with Real Patients: Before deploying AI systems, healthcare organizations should conduct usability testing with actual patients to ensure explanations are comprehensible and actionable, not just technically accurate
• Document Performance Across Populations: Clearly report how AI system accuracy varies across different demographic groups, insurance statuses, and geographic regions so patients understand potential limitations
• Establish Clear Confidence Thresholds: Define what level of AI confidence triggers different clinical actions, and communicate these thresholds to patients in plain language rather than technical metrics
• Train Clinical Staff on AI Communication: Invest in educating doctors and nurses on how to discuss AI recommendations with patients in ways that acknowledge uncertainty and preserve patient autonomy

The challenge extends beyond individual patient conversations. Healthcare systems need institutional infrastructure to support these discussions. This means allocating time in clinical workflows for AI-related conversations, developing communication templates that have been tested for comprehension, and training staff to navigate questions about algorithmic decision-making .

The stakes are significant. As AI-driven diagnostics labs expand globally and become standard in medical imaging and other high-risk applications, the opacity of these systems affects millions of patients. Without meaningful transparency, patients cannot exercise informed consent. Without clear explanations, they cannot understand the reasoning behind treatment recommendations or participate meaningfully in their own care decisions .

The EU AI Act and similar regulations provide the legal framework, but they cannot alone solve the underlying problem. What's needed is a sustained commitment from healthcare organizations, technology developers, and policymakers to prioritize patient understanding alongside algorithmic accuracy. Until that happens, the gap between patients' legal right to understand AI and their actual ability to do so will continue to widen.