OpenAI's newest model GPT-5 hallucinates less than its predecessors, but a parallel study reveals a deeper problem: users follow AI recommendations roughly 80% of the time regardless of accuracy, undermining the benefits of improved model performance. While GPT-5 reduces false claims to 9.6% when given web access (down from 12.9% for GPT-4o), research from the University of Pennsylvania found that people adopt AI advice at nearly the same rate whether the advice is correct or dangerously wrong .

How Much Better Is GPT-5 Really at Avoiding Mistakes?

OpenAI released detailed performance metrics in GPT-5's system card, comparing the model against earlier versions and its own reasoning-enhanced variant, GPT-5-thinking. The results show meaningful progress on factual accuracy when the model can search the web for information .

• With Web Access: GPT-5 makes incorrect claims 9.6% of the time, compared to 12.9% for GPT-4o, representing a 26% reduction in hallucination rates
• With Reasoning Power: GPT-5-thinking drops to just 4.5% hallucination rate, significantly outperforming the reasoning model o3 at 12.7%
• Major Factual Errors: GPT-5 produced 44% fewer responses containing at least one major factual error compared to earlier models

However, these improvements evaporate when the model operates without internet access. On OpenAI's internal Simple QA benchmark, which tests factual knowledge without web browsing, GPT-5's hallucination rate jumps to 47%, nearly matching o3 at 46% . This gap reveals a critical limitation: the model's apparent accuracy depends heavily on external information sources rather than genuine improvement in reasoning.

Why Do Users Trust AI Even When It's Demonstrably Wrong?

A new study from the University of Pennsylvania uncovered a troubling behavioral pattern that may completely undermine the benefits of GPT-5's improved accuracy. Researchers Steven Shaw and Gideon Nave conducted a series of experiments with over 350 participants to understand what they call "cognitive surrender" .

The findings were striking: when participants were given access to ChatGPT to answer reasoning and knowledge-based questions, they followed the AI's correct advice 92.7% of the time. But crucially, they also followed the AI's incorrect advice 79.8% of the time. This means that even when the model gave them the wrong answer, roughly four out of five users who engaged with the AI still adopted its recommendation .

"We felt that the ability to actually outsource thinking hadn't really been studied itself. It's sort of a profound idea. With these AI tools that are available, they're so ingrained in our daily lives and decision processes that we now have the option or ability to outsource thinking itself," said Steven Shaw.

Steven Shaw, Postdoctoral Researcher at University of Pennsylvania

The research suggests that users are not simply making mistakes; they're experiencing a shift in how they process information. When AI presents a plausible-sounding answer, people tend to accept it without engaging their own critical judgment. Over 50% of participants chose to use ChatGPT even when it was optional, suggesting that convenience and perceived authority are driving adoption regardless of accuracy concerns .

The Real-World Cost of Trusting AI Too Much

The practical implications are significant. OpenAI itself acknowledged that GPT-5 is being promoted for healthcare applications, yet one in 10 responses still contain hallucinations. In a medical context, this error rate could have serious consequences .

The problem became visible almost immediately after GPT-5's launch. Beth Barnes, founder and CEO of the AI research nonprofit METR, spotted a factual error in OpenAI's official demo of GPT-5 explaining how airplane wings work. The model cited a common misconception related to the Bernoulli Effect, demonstrating that even high-profile demonstrations can contain the very errors the company claims to have reduced .

"The capacity to think critically, the capacity to be able to check what the AI is giving you has become more and more important over time. This is kind of a muscle that we have, that hopefully we are not going to lose over time," noted Gideon Nave.

Gideon Nave, Marketing Professor at University of Pennsylvania

Researchers worry that as AI becomes more integrated into daily life, people may lose the ability to verify information independently. The Pennsylvania study found that participants adopted incorrect AI answers and became more confident in those answers, suggesting that exposure to AI doesn't just change behavior; it changes how people feel about their own decision-making .

How to Verify AI Recommendations Before Acting on Them

• Enable Web Search: When using ChatGPT or similar tools, activate web browsing features to reduce hallucination rates from 47% to under 10%, as GPT-5's performance data shows
• Cross-Reference Critical Information: For healthcare, financial, legal, or safety-related advice, independently verify AI recommendations through authoritative sources before making decisions
• Question Plausible-Sounding Answers: Train yourself to pause when AI provides confident-sounding responses, especially on specialized topics where errors could have real consequences
• Use AI as a Starting Point, Not a Conclusion: Treat AI outputs as initial research or brainstorming rather than final answers, particularly for decisions that affect your health, finances, or safety

The gap between GPT-5's improved accuracy metrics and users' actual behavior reveals a fundamental challenge in AI adoption. Better models alone won't solve the problem if people aren't equipped to recognize when those models fail. As AI becomes more capable and more trusted, the responsibility falls on both developers to be transparent about limitations and on users to maintain healthy skepticism .