What It Actually Means
The word is unhelpful because it suggests a malfunction, something that occasionally goes wrong and could be fixed. That is not what is happening.
A language model generates text by predicting what should plausibly come next. It has no separate store of facts to check against and no internal sense of certainty. Producing a true statement and producing a convincing false one are the same operation. When it is right, that is because plausible and true coincided, which they usually do, and that is exactly why the exceptions are so hard to spot.
Why It Will Not Be Fixed
This matters when a supplier tells you the next model will solve it.
Newer models hallucinate less. They will not stop, because the behaviour is inseparable from the mechanism that makes them useful. The same willingness to produce a fluent answer from incomplete information is what lets one draft an email from a one-line instruction.
What has genuinely improved is a related thing: better models are more willing to say they do not know. That helps and it is not the same as accuracy.
What This Means Commercially
The practical consequence is that fluency has become worthless as a signal of correctness, and people are extremely bad at ignoring it. A confident, well-structured, correctly formatted wrong answer is far more dangerous than an obviously broken one, because nothing about it invites checking.
That reframes the question from how accurate the AI is to a better one: what happens when it is wrong, and who finds out?
Some jobs absorb the error easily. Drafting something a person will edit anyway, summarising a document that person will still read, suggesting categories somebody confirms. The error surfaces immediately and costs a moment.
Others do not. Anything that goes to a customer unread, feeds a calculation, drives a decision nobody revisits, or states a policy, a price or an obligation. Here a hallucination becomes a commitment you did not intend to make, and you may find out from the customer.
What To Ask
- What happens when it is wrong, and who notices? The single most useful question about any AI proposal.
- Does anything go to a customer without a person reading it? If yes, that is a decision worth making deliberately.
- Is it answering from our data or from what it absorbed in training? Grounding it in your own content through RAG reduces invention substantially. It does not remove it.
- Can it say it does not know? A system with no route to escalate will always produce something.
Designing Around It
The reliable pattern is not trying to prevent hallucination. It is confining the system to work where being wrong is cheap and visible.
Give it a defined body of material to answer from and require it to cite where an answer came from, so a person can check in seconds. Have it draft rather than send. Define what it must refuse and route those cases to a person through a proper handoff. Keep the human as the last step wherever the output carries a commitment.
Done that way, these systems are genuinely valuable. The failures we see are almost never a model being unusually bad. They are a model being handed a job where nobody was going to check.
More terms are in the glossary.
