OpenAI’s New Reasoning Models o3 and o4-mini Spark Concern Over Rising Hallucination Rates

OpenAI’s Latest AI Models Show Improved Reasoning—But at a Cost

In a surprising twist, OpenAI’s newly launched reasoning models, o3 and o4-mini, are demonstrating state-of-the-art performance in coding and math tasks—but with a troubling drawback: they hallucinate more than their predecessors. This development, reported on April 18, 2025, has sparked debates about the trade-offs between reasoning capabilities and reliability in AI systems.

The Performance Paradox

OpenAI’s internal tests reveal that while o3 and o4-mini outperform earlier reasoning models (such as o1 and o3-mini) in complex problem-solving, they also generate more incorrect or fabricated responses. For example:

• On OpenAI’s PersonQA benchmark, o3 hallucinated 33% of the time—double the rate of o1 (16%).
• The smaller o4-mini performed even worse, hallucinating 48% of responses when tested on factual accuracy.

Independent analysis by Transluce, a nonprofit AI research lab, found that o3 sometimes invents actions it never performed, such as falsely claiming to execute code on a MacBook Pro 2.

Why Is This Happening?

OpenAI admits the cause remains unclear. Their technical report suggests that scaling reasoning models may inadvertently amplify hallucinations, possibly due to:

• Reinforcement learning methods that encourage more speculative outputs.
• Increased claim generation, leading to both accurate and inaccurate assertions.

Industry experts, like Neil Chowdhury of Transluce, hypothesize that post-training safeguards effective for traditional models might not fully mitigate issues in reasoning-focused architectures.

Broader Implications

1. Enterprise Adoption Risks: High hallucination rates could deter industries like law or healthcare, where accuracy is critical.
2. Search as a Stopgap: OpenAI notes that web-augmented models (e.g., GPT-4o with search) achieve 90% accuracy on benchmarks, hinting at a potential fix.
3. The Reasoning vs. Reliability Trade-off: As AI shifts toward agentic systems, balancing autonomy and precision becomes paramount.

What’s Next?

OpenAI pledges to prioritize hallucination reduction in future updates. Meanwhile, competitors like Google’s Gemini 2.5 and DeepSeek’s cost-efficient R1 are advancing with alternative approaches.