Hallucinations Are Architecture, Not a Bug: Why LLMs Confidently Lie

In June 2023, a New York lawyer cited six federal cases that did not exist.

The cases were on letterhead, properly formatted, with plausible judges and jurisdictions. ChatGPT had invented every one of them, and the resulting sanction in Mata v. Avianca made the front page of the New York Times.

The reflex is to call this a bug. It is not. Hallucination — the technical term for an LLM (large language model — the kind of system behind ChatGPT, Claude, Gemini) confidently producing fluent text that happens to be false — comes from the same machinery that lets the model write at all. This guide explains why, what reduces it, and what you should and should not ask one of these things to do.

What a hallucination actually is

A hallucination is not a lie. The model does not know the truth and conceal it. It generates output that looks identical to a true statement — same syntax, same tone, same air of authority — and which happens, in this case, not to correspond to anything in the world. The model has no internal flag for "this part is real, this part is invented."

The cleanest examples are citations. Ask any frontier model for sources on an obscure topic and it will return author, title, year, journal, page numbers — beautifully formatted, often wrong. Lawyers, journalists, and graduate students have learned this the expensive way. The same pattern shows up in code generation as "package hallucination" — models confidently importing software libraries that have never existed, which by some estimates appears in a meaningful fraction of code suggestions across major models. The package looks real. The install fails.

The defining property is confidence. A hedged answer ("I'm not sure, but possibly…") is not a hallucination. A wrong fact, delivered in the same tone as a right one, is. The model is not unreliable in the human sense of being sloppy. It is uniformly fluent, which makes the wrong outputs harder to spot than they would be coming from a person.

Why it happens: a model that has never read a fact

To see why hallucinations are inevitable, look at what an LLM actually does. A transformer — the neural-network architecture introduced in Vaswani et al.'s 2017 paper Attention Is All You Need, which underlies every modern LLM from GPT-4 to Claude 3 to Llama 3 — takes a sequence of tokens (rough chunks of text, typically a few characters each) and predicts the next one. That is the entire job. Repeat a few thousand times and you have a paragraph.

During training, the model sees trillions of tokens scraped from the internet, books, code, and papers. It does not store these as facts in any retrievable sense. It adjusts billions of internal weights so that, given any plausible prefix, the next token is the one that statistically tends to follow. There is no database. There is no truth check. There is a vast statistical surface that, when sampled, produces text that resembles its training distribution.

When you ask such a system "what is the capital of France," it generates "Paris" because in essentially every text where that question appears, "Paris" follows. When you ask for an obscure 1987 Ninth Circuit ruling on maritime liability, it generates something that looks like a 1987 Ninth Circuit ruling on maritime liability, because that is what follows the prompt. The model has no mechanism to distinguish "this answer is an indexed fact" from "this answer is a plausible shape." Both are the same operation. The first one happens to land on something true; the second does not.

Creativity and hallucination are the same circuit

This is the part most coverage misses. The capacity to write a passable sonnet about a cat depends on the same architecture as the capacity to invent a federal case. Both are the model generating tokens that the training distribution suggests would plausibly follow the prompt. In neither case does the model "know" anything; it is producing fluent shape.

A model that could not hallucinate could not write fiction. It could not summarize a paper it has not seen verbatim, propose code for a problem unlike any in training, or rephrase your email in three different registers. Stochastic recombination — sampling from a probability distribution rather than retrieving from an index — is the feature. The price is that the same machinery, pointed at a factual question outside its training, produces fluent nonsense with the same conviction it produces a sonnet.

This is why Google's Bard, in its February 2023 launch demo, claimed the James Webb Space Telescope took the first image of an exoplanet. (It did not; the first direct image of an exoplanet was made by the VLT/NACO instrument in 2004.) Alphabet's stock fell roughly 9% on the day, by some estimates erasing $100 billion in market cap. Bard was not malfunctioning. It was working exactly as designed — the same design that, two months later, would write a usable draft of a quarterly summary when someone at Google asked it to.

The interesting question is not "why do they hallucinate." It is "why are we surprised when a system trained to produce plausible text produces plausible text."

What helps, what doesn't, and what never fully will

Several techniques meaningfully reduce the hallucination rate. It is worth knowing which, and at what cost.

RAG — retrieval-augmented generation, where the model is given relevant documents at inference time and instructed to answer from them — is the dominant approach in production. Instead of asking GPT-4 "what does our company's leave policy say," you fetch the policy first and paste it into the prompt. The model still generates fluently, but it generates against grounded text rather than from training memory. Hallucination rates drop, sometimes by an order of magnitude. They do not go to zero. The model can still misread the document, conflate two passages, or ignore the instruction to stay grounded.

Tool use — letting the model call a calculator, a web search, a code interpreter, a database — works on the same principle: outsource the part of the answer that requires ground truth. Anthropic's Claude and OpenAI's GPT-4 both ship with this. It works well for arithmetic, recent events, and structured lookups; less well for synthesis questions where the tool's output still has to be summarized in prose, which is where a fresh round of confabulation can creep in.

Training-data filtering and RLHF (reinforcement learning from human feedback — the process by which human raters teach the model which answers are preferred) both reduce the rate at which a model confidently invents. They do not eliminate it. A model can be trained to say "I don't know" more often, but the same training also makes it say "I don't know" to questions it could have answered correctly. There is a frontier between confidence and accuracy that no amount of fine-tuning fully erases — push one and the other moves.

The deeper limit is architectural. As long as the model is a probability distribution over token sequences, with no explicit fact-retrieval step, it will sometimes sample a token that does not correspond to reality. Yann LeCun, Meta's chief AI scientist, has been the loudest public voice arguing that hallucination will only be solved by architectures fundamentally different from transformers — systems that reason in structured representations rather than at the level of surface tokens. Whether that is two years away or twenty is, charitably, contested.

What this means for you

If you use LLMs at work, the practical rule is short: never trust a fact you cannot verify, and especially never trust a citation. Quotes, author names, case numbers, chemical formulas, API endpoints, statutes, dates of events more than a few years old — these are the model's most dangerous output, because they are the most authoritative-looking and the easiest to fabricate. Treat any factual claim from an LLM the way a journalist treats a tip from a source: useful as a starting point, useless until corroborated.

If you build with them, scope is the lever. A model asked to summarize a document you provide will hallucinate far less than one asked to recall a fact. Constrain the task to what the model can see, and put a verification step — retrieval, a database lookup, a deterministic check — on anything that touches reality. The applications working in production right now — Microsoft Copilot for Office, Harvey for legal, GitHub Copilot for code — all do this. The ones that fail are the ones that ask the model to know.

If you are reading this as a citizen of a country whose institutions are starting to use these systems for medical triage, hiring, or legal advice, the load-bearing fact to internalize is this: the model is not malfunctioning when it invents a case. That is the technology. The question is not how to make hallucinations go away but which decisions can tolerate them and which cannot. The press release will not tell you. You have to ask.

Ask Gradient

Want to dig further? Try a specific follow-up on Gradient's Ask Gradient — for example, "How does retrieval-augmented generation actually reduce hallucination, and where does it still fail?" Ask Gradient answers from a curated corpus of AI research and reporting rather than from training memory, which is a working example of the principle this guide just described: ground the model, scope the question, verify the citation.