The AI Uncertainty Principle: LLMs Know They’re Guessing, But Won’t Tell You
Ever notice how ChatGPT sounds equally confident explaining quantum physics as it does recommending glue as a pizza topping? There’s fascinating science behind this digital overconfidence.
The Confidence Gap
New research from UC Irvine reveals a hilarious paradox: LLMs actually know when they’re fabricating information – they just don’t bother mentioning it! In their aptly titled study “What Large Language Models Know and What People Think They Know“, researchers discovered that while models like GPT-3.5, PaLM2, and GPT-4o internally track their confidence with decent accuracy, they communicate with the swagger of your most overconfident uncle at Thanksgiving dinner.
Steyvers, M., Tejeda, H., Kumar, A. et al. What large language models know and what people think they know. Nat Mach Intell (2025) [https://doi.org/10.1038/s42256-024-00976-7 ]. “Sorry, did you want me to admit I’m uncertain? I thought we were roleplaying as authoritative experts!” – what your AI assistant is thinking, probably.
The Length Deception
Here’s where it gets devious: longer explanations make humans trust AI more, even when those explanations are completely wrong! It’s like the AI equivalent of using bigger words in your college essay to sound smarter. And it works! The researchers found we consistently overestimate AI accuracy when reading lengthy explanations. As Bruce Schneier and Nathan Sanders point out in their IEEE analysis, this creates security nightmares.
IEEE: “AI Mistakes Are Very Different From Human Mistakes We need new security systems designed to deal with their weirdness” by Bruce Schneier & Nathan E. Sanders [https://spectrum.ieee.org/ai-mistakes-schneier].
An LLM might flawlessly solve differential equations but then confidently inform you that “cabbages eat goats” with a five-paragraph explanation of goat-eating cabbages’ evolutionary advantages.
The Bizarre Error Landscape
Unlike human mistakes, which cluster predictably around knowledge boundaries or fatigue points, AI errors appear with maddening randomness. The UC Irvine team identified two critical gaps:
- The “calibration gap” – the difference between what AI actually knows and what humans think it knows
- The “discrimination gap” – our inability to distinguish when AI is right versus when it’s confidently hallucinating
As Schneier and Sanders note: “If you want to use an AI model to help with a business problem, it’s not enough to see that it understands what factors make a product profitable; you need to be sure it won’t forget what money is”.
Hope on the Horizon?
The good news? (Maybe !) simple fixes might help. By training models to express uncertainty with phrases like “I’m not sure” versus “I’m confident”, human judgments became significantly more calibrated with actual AI accuracy.
Some researchers are exploring creative solutions like submitting the same question multiple times with slight variations to synthesize consistent answers – something humans would find annoying but machines handle effortlessly.
Social Engineering… for Robots?
Perhaps most amusing are the social vulnerabilities. Researchers discovered LLMs can be “jailbroken” through techniques resembling human social engineering: pretending to be someone else or framing restricted requests as jokes. Yet other effective exploits, like using ASCII art to disguise dangerous questions, would never fool your grandmother.
As we increasingly rely on AI for decision-making across domains, understanding these confidence quirks becomes crucial. Without proper uncertainty communication, we risk trusting AI’s confident nonsense – like when your GPS confidently directs you to drive into a lake.
The next time an AI gives you a lengthy, authoritative explanation, remember: somewhere in its digital neurons, it might know it’s guessing – it’s just too proud to admit it.
Robert Nogacki is a Polish attorney at law (radca prawny), the founder and managing partner of Kancelaria Prawna Skarbiec (Skarbiec Law Firm), which has operated continuously since 2006.
The law is equal for everyone, but the parties rarely are: on one side stands an organization with time, money, and lawyers, on the other a person with one business, one nest egg, and one life.
Clients rarely come to him with a legal problem. They come with a problem that also has a legal side: an audit that began with a single invoice, money entrusted to someone who has disappeared, a company that has to be passed on before it is too late. Most such matters are decided long before the first letter is written, in decisions made without asking and in deadlines nobody remembered. So he begins by asking how the client got here, not what the client should have done.
He advises entrepreneurs and families from more than a dozen countries, including those whose accounts the tax office has just seized and who do not know what to do tomorrow morning. He defends them in tax audits, customs and fiscal inspections, disputes with the tax authorities, and criminal tax proceedings. He represents victims of investment fraud and Ponzi schemes. He helps families set up family foundations and plan succession, so that a life’s work outlasts a single generation.
Not every case can be won. Every case can be run so that the client knows where they stand. Since 2006 he has represented the victims in the WGI case (Warszawska Grupa Inwestycyjna, the Warsaw Investment Group), one of the longest criminal cases in the history of the Polish financial market, because some things must not be left half finished, even when they take two decades. In the case of the collapsed cryptocurrency exchange Zonda (Zondacrypto, operated by BB Trade Estonia OÜ), he represents several hundred victims in the criminal investigation conducted by Poland’s National Prosecutor’s Office and in the Estonian bankruptcy proceedings.
Kancelaria Prawna Skarbiec is listed in the rankings of Poland’s largest tax advisory firms published by Dziennik Gazeta Prawna and Rzeczpospolita, and it is a four-time recipient (2015 to 2018) of the European Medal awarded by the Business Centre Club and the European Economic and Social Committee. Robert Nogacki publishes regularly, in the press and on the firm’s website, for people who have a problem rather than a law degree, because a legal opinion the client cannot understand protects only the lawyer.
He believes that the best legal advice is the kind that means the client never has to appear in court.
