An international team of researchers led by MIT is developing a new approach to medical AI that prioritizes humility over certainty, designing systems that openly acknowledge when they lack confidence in a diagnosis. The framework addresses a critical problem: current AI systems often steer doctors toward incorrect decisions because they present recommendations with unwarranted confidence, even when the evidence doesn't support such certainty. What Happens When AI Systems Are Too Confident? Previous studies have shown that physicians in intensive care units defer to AI systems they perceive as reliable, even when their own clinical intuition contradicts the AI's recommendation. Both doctors and patients are more likely to accept incorrect AI suggestions when they appear authoritative. This dynamic creates a dangerous situation where the very confidence that makes AI seem trustworthy can actually lead to medical errors. The MIT-led consortium, which includes researchers from the University of Melbourne and other institutions, has created a framework with computational modules designed to address this problem. The centerpiece is something called the Epistemic Virtue Score, which acts as a self-awareness check for AI models. This module requires the AI to evaluate its own certainty when making diagnostic predictions, ensuring that confidence levels are appropriately matched to the actual evidence available in each clinical scenario. "We're now using AI as an oracle, but we can use AI as a coach. We could use AI as a true co-pilot. That would not only increase our ability to retrieve information but increase our agency to be able to connect the dots," said Leo Anthony Celi, a senior research scientist at MIT's Institute for Medical Engineering and Science. Leo Anthony Celi, Senior Research Scientist, MIT Institute for Medical Engineering and Science How Does Humble AI Actually Work in Practice? When an AI system detects that its confidence exceeds what the available evidence supports, it pauses and flags the mismatch. Rather than pushing forward with a diagnosis, the system can request specific tests or medical history that would resolve the uncertainty, or recommend specialist consultation. The goal is an AI that provides answers while also signaling when those answers should be treated with caution. Sebastián Andrés Cajas Ordoñez, lead author of the study published in BMJ Health and Care Informatics, explained the philosophy behind this approach: "We are trying to include humans in these human-AI systems, so that we are facilitating humans to collectively reflect and reimagine, instead of having isolated AI agents that do everything. We want humans to become more creative through the usage of AI," stated Sebastián Andrés Cajas Ordoñez, researcher at MIT Critical Data. Sebastián Andrés Cajas Ordoñez, Researcher, MIT Critical Data Ways to Implement Humble AI in Healthcare Settings - Self-Awareness Modules: Integrate computational systems that evaluate the AI's own confidence levels and flag when certainty exceeds available evidence, preventing overconfident recommendations. - Collaborative Design: Involve doctors, data scientists, patients, and social scientists in designing AI systems from the ground up, ensuring the tools are built for and by the people who will use them. - Uncertainty Signaling: Program AI to request additional tests, medical history, or specialist consultation when diagnostic uncertainty is high, rather than pushing forward with potentially incorrect recommendations. - Diverse Data Sources: Train AI systems on inclusive datasets that represent multiple populations and perspectives, reducing bias toward particular ways of thinking about medical issues. The MIT team is currently implementing this framework into AI systems based on the MIMIC database, a large-scale collection of medical records from Beth Israel Deaconess Medical Center. They are introducing these systems to clinicians in the Beth Israel Lahey Health system. The approach could also be applied to AI systems used for analyzing X-ray images or determining treatment options in emergency rooms. Why Consumer Adoption of Health AI Is Accelerating Faster Than Trust? While researchers work to make AI more trustworthy, consumer adoption of AI for health and self-care has doubled in just one year. According to a survey from Rock Health conducted in December 2025, 32 percent of U.S. consumers used AI chatbots for health information in 2025, up from 16 percent in 2024. ChatGPT, owned by OpenAI, was the most popular brand, used by 23 percent of consumers seeking health information, followed by Google's Gemini at 15 percent. However, trust in AI for health decisions lags behind usage. An Employee Benefit Research Institute survey found that 55 percent of working-age Americans trust AI and digital tools less than they do care from a health provider. At the same time, 47 percent of consumers trust the health care decisions that AI provides. This gap between adoption and trust suggests that many people are using AI for health guidance without full confidence in its reliability. When consumers do use AI for health purposes, they typically follow up with multiple actions. The most common responses include searching for more information (42 percent), consulting a provider (40 percent), discussing findings with family or friends (35 percent), and trying out a new health behavior (32 percent). Among those using AI for health, 59 percent explored treatment options based on a diagnosis, and 55 percent researched prescription drugs or side effects. Notably, 42 percent of consumers expressed a desire to use AI tools to help make health care decisions or choose health plans, but said they don't know where to start. This represents both a challenge and an opportunity for the healthcare industry to provide clearer guidance on how to use AI responsibly. The MIT research suggests that the solution lies not in abandoning AI for medical diagnosis, but in fundamentally redesigning how these systems interact with doctors. By creating AI that is transparent about its limitations and collaborative in its approach, researchers hope to harness the technology's potential while protecting against the risks of overconfidence. As these humble AI systems move from research into clinical practice, they may help bridge the gap between the rapid adoption of health AI and the slower growth of trust in its recommendations.
