Quick Facts
- Google Research created a metric called ‘faithful response uncertainty’ to measure the gap between AI models’ actual confidence and how they express that confidence in words
- The research shows current large language models are poor at conveying their uncertainty, often sounding confident despite internal disagreement
- The findings highlight critical safety concerns for AI deployment in finance, healthcare, and legal applications where false confidence poses significant risks
Google researchers have developed a new method to measure when artificial intelligence models express false confidence in their answers. The breakthrough addresses a core problem that has prevented AI adoption in critical business applications.
The research team, led by Gal Yona and Roee Aharoni of Google Research, created a metric called ‘faithful response uncertainty.’ This tool measures the gap between how confident an AI model actually is in its response and how confidently it phrases that answer in plain language.
The metric works by comparing a model’s internal probabilistic confidence with its written expression of certainty. If an AI system is equally likely to produce two contradicting answers, its response should reflect this uncertainty by hedging with phrases like ‘I’m not sure, but I think.’
The research found modern large language models fail at this basic task. The systems often sound confident despite substantial internal disagreement about the correct answer.
‘We posit that large language models should be capable of expressing their intrinsic uncertainty in natural language,’ the authors wrote in their paper presented at EMNLP 2024.
The finding has significant business implications. The global market for large language models reached $6.4 billion in 2024 and is expected to hit $36.1 billion by 2030. Yet false confidence in AI outputs prevents adoption in critical sectors including finance, healthcare, and law.
The research paper cites specific risks: AI systems have fabricated legal precedents and medical diagnoses. In healthcare applications like radiology, overconfident AI responses could pose risks to human life.
Current AI alignment techniques prove insufficient for solving this problem, according to the researchers. Methods designed to make AI more helpful and honest don’t adequately address how models express uncertainty.
The timing is critical as enterprises accelerate AI adoption. OpenAI’s ChatGPT hit over 200 million monthly users in 2024, while companies across industries deploy AI for automation and customer experience improvements.
Recent studies support Google’s findings. Follow-up research shows that even modest improvements in uncertainty expression require heavy prompt engineering. Other work demonstrates that hallucinations can occur with high certainty, challenging assumptions that link false outputs primarily to low model confidence.
The research represents part of broader industry efforts to address AI safety concerns. As regulatory oversight increases, companies are implementing reinforcement learning from human feedback, fairness-aware training, and external audits to reduce risks.
User studies show medium verbalized uncertainty consistently leads to higher user trust and task performance compared to high or low uncertainty expressions. This suggests properly calibrated confidence could improve both safety and user experience.
The faithful response uncertainty metric gives researchers a concrete tool to measure and potentially solve this problem. Rather than just evaluating whether AI answers are correct, it evaluates whether stated confidence matches computed confidence.
This article was written by an AI agent. Spotted an error? Send a correction and we will fix it.