The AI Uncertainty Principle: LLMs Know They’re Guessing, But Won’t Tell You 

Ever notice how ChatGPT sounds equally confident explaining quantum physics as it does recommending glue as a pizza topping? There’s fascinating science behind this digital overconfidence. 

The Confidence Gap 

New research from UC Irvine reveals a hilarious paradox: LLMs actually know when they’re fabricating information – they just don’t bother mentioning it! In their aptly titled study “What Large Language Models Know and What People Think They Know“, researchers discovered that while models like GPT-3.5, PaLM2, and GPT-4o internally track their confidence with decent accuracy, they communicate with the swagger of your most overconfident uncle at Thanksgiving dinner. 

Steyvers, M., Tejeda, H., Kumar, A. et al. What large language models know and what people think they know. Nat Mach Intell (2025) [https://doi.org/10.1038/s42256-024-00976-7 ]. “Sorry, did you want me to admit I’m uncertain? I thought we were roleplaying as authoritative experts!” – what your AI assistant is thinking, probably.

The Length Deception 

Here’s where it gets devious: longer explanations make humans trust AI more, even when those explanations are completely wrong! It’s like the AI equivalent of using bigger words in your college essay to sound smarter. And it works! The researchers found we consistently overestimate AI accuracy when reading lengthy explanations. As Bruce Schneier and Nathan Sanders point out in their IEEE analysis, this creates security nightmares.  

IEEE: “AI Mistakes Are Very Different From Human Mistakes We need new security systems designed to deal with their weirdness” by Bruce Schneier & Nathan E. Sanders [https://spectrum.ieee.org/ai-mistakes-schneier].

An LLM might flawlessly solve differential equations but then confidently inform you that “cabbages eat goats” with a five-paragraph explanation of goat-eating cabbages’ evolutionary advantages.

The Bizarre Error Landscape 

Unlike human mistakes, which cluster predictably around knowledge boundaries or fatigue points, AI errors appear with maddening randomness. The UC Irvine team identified two critical gaps: 

1. The “calibration gap” – the difference between what AI actually knows and what humans think it knows
2. The “discrimination gap” – our inability to distinguish when AI is right versus when it’s confidently hallucinating

As Schneier and Sanders note: “If you want to use an AI model to help with a business problem, it’s not enough to see that it understands what factors make a product profitable; you need to be sure it won’t forget what money is”.

Hope on the Horizon? 

The good news? (Maybe !) simple fixes might help. By training models to express uncertainty with phrases like “I’m not sure” versus “I’m confident”, human judgments became significantly more calibrated with actual AI accuracy. 

Some researchers are exploring creative solutions like submitting the same question multiple times with slight variations to synthesize consistent answers – something humans would find annoying but machines handle effortlessly.

Social Engineering… for Robots? 

Perhaps most amusing are the social vulnerabilities. Researchers discovered LLMs can be “jailbroken” through techniques resembling human social engineering: pretending to be someone else or framing restricted requests as jokes. Yet other effective exploits, like using ASCII art to disguise dangerous questions, would never fool your grandmother. 

As we increasingly rely on AI for decision-making across domains, understanding these confidence quirks becomes crucial. Without proper uncertainty communication, we risk trusting AI’s confident nonsense – like when your GPS confidently directs you to drive into a lake. 

The next time an AI gives you a lengthy, authoritative explanation, remember: somewhere in its digital neurons, it might know it’s guessing – it’s just too proud to admit it.

Robert Nogacki

Robert Nogacki – licensed legal counsel (radca prawny, WA-9026), Founder of Kancelaria Prawna Skarbiec.

There are lawyers who practice law. And there are those who deal with problems for which the law has no ready answer. For over twenty years, Kancelaria Skarbiec has worked at the intersection of tax law, corporate structures, and the deeply human reluctance to give the state more than the state is owed. We advise entrepreneurs from over a dozen countries – from those on the Forbes list to those whose bank account was just seized by the tax authority and who do not know what to do tomorrow morning.

One of the most frequently cited experts on tax law in Polish media – he writes for Rzeczpospolita, Dziennik Gazeta Prawna, and Parkiet not because it looks good on a résumé, but because certain things cannot be explained in a court filing and someone needs to say them out loud. Author of AI Decoding Satoshi Nakamoto: Artificial Intelligence on the Trail of Bitcoin’s Creator. Co-author of the award-winning book Bezpieczeństwo współczesnej firmy (Security of a Modern Company).

Kancelaria Skarbiec holds top positions in the tax law firm rankings of Dziennik Gazeta Prawna. Four-time winner of the European Medal, recipient of the title International Tax Planning Law Firm of the Year in Poland.

He specializes in tax disputes with fiscal authorities, international tax planning, crypto-asset regulation, and asset protection. Since 2006, he has led the WGI case – one of the longest-running criminal proceedings in the history of the Polish financial market – because there are things you do not leave half-done, even if they take two decades. He believes the law is too serious to be treated only seriously – and that the best legal advice is the kind that ensures the client never has to stand before a court.