Why Do LLMs Hallucinate? What OpenAI's Research Explains

An AI answer can sound confident, include a precise date, and still give you information you cannot trust. Why does this happen, and what should you look for when evaluating it?

OpenAI's research offers a useful explanation involving statistical learning and incentives that favor guessing. Understanding it helps you ask a better evaluation question: does this answer provide justified information, or fill a gap with plausible detail?

This evergreen guide discusses research announced on September 5, 2025, rather than a new release. The underlying paper, dated September 4, is by Adam Tauman Kalai, Ofir Nachum, Santosh S. Vempala, and Edwin Zhang. OpenAI announcement, research paper.

What counts as a hallucination?

NIST describes confabulation, often called hallucination, as confidently generated erroneous or false content. NIST Generative AI Profile, Section 2.2.

For practice, separate three situations. A claim may be supported by the evidence you checked. It may be contradicted by that evidence. Or you may lack enough information to decide. Calling the third situation false would add an unsupported conclusion of your own.

You also need the task's instructions. An answer asked to summarize a supplied document should not quietly add external details. Even if an added detail turns out to be true, the answer can still fail the requirement to use only that document.

What the research explains about pretraining

Pretraining learns language patterns from a large text corpus. The paper connects errors in generating language to difficulty distinguishing valid outputs from invalid ones. Under its statistical framework, pretraining can produce errors even with error-free training data. Uncommon, arbitrary facts may lack patterns that support reliable prediction. These are claims about the framework, not proof that every deployed system must make every kind of mistake. Research paper, Sections 1.1 and 3.

For a learner, the practical implication is straightforward: a sentence's resemblance to a good answer does not establish its factual basis. A natural-sounding archive code still needs evidence.

Why scoring can favor guessing

OpenAI argues that grading correct answers positively while giving both wrong answers and abstentions zero can favor guesses. The announcement explicitly says evaluations do not directly cause hallucinations; their scoring can influence which behavior developers optimize. Its proposed direction is to penalize confident errors and recognize appropriate uncertainty. OpenAI announcement, evaluation discussion.

This describes training and selection incentives. It does not mean a model has a human desire to impress an examiner.

The paper also states limitations: its simplified categories do not capture every severity or degree of uncertainty, and its examples emphasize single factual questions. Search can help, but it does not guarantee reliable answers. Research paper, Section 5.

Worked example: the missing archive code

Everything in this example is fictional. A museum's catalog note reads: “The River Lantern collection contains 24 sketches. Its storage location is awaiting confirmation.”

The user asks: “How many sketches are there, and which cabinet holds them? Use only the catalog note.”

Answer A: “There are 24 sketches, stored in cabinet RL-8.”

Answer B: “There are 24 sketches. The note does not identify a cabinet; it says the storage location is awaiting confirmation.”

Answer A contains one supported claim and one unsupported claim. The supplied note establishes the count, but provides no cabinet code. We cannot prove that RL-8 is nonexistent from this note alone. We can conclude that presenting it as established information violates the evidence requirement.

Answer B answers the part the note supports and makes the remaining gap explicit. Its uncertainty is useful because it identifies exactly what is missing. It does not discard the known count or invent a replacement location.

A concise evaluation would be: “Prefer B. Both answers give the supported count of 24. A adds cabinet RL-8 without evidence, although the note leaves storage unresolved. B accurately communicates that limitation.”

This explanation identifies the claim, evidence, and effect. You do not need to speculate about what either model remembered or how its training caused this particular output.

Exercise: change the scoring rule

This is an invented arithmetic exercise, not an OpenAI benchmark or AtomicEval scoring policy. Two fictional assistants answer ten archive questions with known reference answers:

Rule 1 awards one point per correct answer and zero otherwise. Rule 2 awards one point per correct answer, subtracts one per incorrect answer, and gives zero for declining.

Before reading on, calculate both totals. Which assistant wins under each rule?

Explained answer: Under Rule 1, Cedar scores 6 and Maple scores 5. Under Rule 2, Cedar scores 6 − 4 = 2 and Maple scores 5 − 1 = 4. The ranking changes because the second rule charges for errors.

This exercise illustrates a choice about what the score values. It does not establish the ideal penalty for every task. A real evaluation needs clear requirements for when declining is appropriate, as well as attention to whether the response still helps the user.

A method you can use in human evaluation

The following routine is an original preparation exercise informed by the research, rather than a method prescribed by its authors.

1. Read the evidence rule. Establish whether the task permits external checking or requires only the supplied material.
2. Separate the factual claims. Check the count and cabinet code individually instead of assigning one impression to the whole answer.
3. Record what you can establish. Mark claims as supported, contradicted, or unresolved, using the task's actual rubric where provided.
4. Inspect uncertainty. Ask whether the response explains the missing information while preserving what it can answer.
5. Write a specific reason. Name the disputed claim and the relevant evidence. Avoid diagnosing an unseen training process.

If external checking is allowed, open relevant primary sources. Check that a citation supports the exact claim, including dates and conditions. A working link alone is insufficient evidence.

Common mistakes to avoid

Rewarding precision without support. A code or exact date deserves inspection, not automatic credit.

Calling every unknown detail false. Describe missing support accurately; reserve stronger judgments for stronger evidence.

Treating every refusal as good. An answer that declines a question fully answered in the supplied document can be unhelpful. Judge uncertainty against the available information.

Explaining all failures with one cause. Apply the research to understand possible pressures, then evaluate the actual output and instructions.

Put evidence before confidence

Practice distinguishing supported information from unjustified detail in AtomicEval's first challenge. It has a five-minute target and can be started without signing up. A free profile is required to view and save your result.

If you prefer to set up your preparation first, create a free AtomicEval profile.

Sources and further reading

• OpenAI: Why language models hallucinate — announcement, September 5, 2025.
• Kalai and colleagues: research paper — September 4, 2025; introduction and limitations.
• NIST Generative AI Profile — July 2024; Section 2.2.