In 2023 a lawyer in New York handed a court a document listing six earlier cases. The airline's lawyers could not find them. Nor could the judge. None of the six existed. The lawyer had asked a chatbot for cases, and it had written six, with names, dates and quotations, in the voice of a law book.
Billions of pages→The shape of good text→Next word, then the next→Something that reads right
A chatbot is a model, lesson 1, trained on an enormous pile of text to guess the next word. It learned what a law case looks like: the shape of the name, the year in brackets. It did not learn a list of real cases to check against. Asked for cases, it made text shaped like cases.
The chatbot
↓→
Six casesshaped like cases
The chatbot wrote six cases that did not exist. Which best describes what went wrong?
Nothing, from the model's side. It made text shaped like cases, which is what it was built to do.
Right. The model was asked for case-shaped text and produced it. Nothing in it knows which cases exist. The failure was in treating the shape as a check.
It lied. It knew the cases were fake.
Not yet. Lying needs knowing. The model has no list of real cases to know against. It has the shape of a case, and it filled the shape.
It was hacked.
Not yet. Nobody broke in. This is the ordinary behaviour of a model that guesses the next word, on a question where the true next words were not in its pile.
An everyday fact95 true in 100
A rare fact65 true in 100
An exact quotation40 true in 100
Made up and round. How often the answer is true.
People call this hallucination. The word just means the model wrote something that reads true and is not. It happens most where the pile of text was thinnest: rare facts, exact numbers, quotations, references. Where the pile was thick, the guess is usually right. Where it was thin, the guess is still fluent.
Move the control from an everyday question to one nobody has written about. The figures are made up and round.
Reads well, out of 1010
Is true, in 10095
How rare the question is: Everyday
Everyday: the answer reads well, 10 out of 10, and is true about 95 times in 100. The pile of text was thick here, so the next-word guesses mostly land on facts.
Reads well, out of 1010
Is true, in 10085
How rare the question is: Uncommon
Uncommon: the answer reads well, 10 out of 10, and is true about 85 times in 100. The pile of text was thick here, so the next-word guesses mostly land on facts.
Reads well, out of 1010
Is true, in 10065
How rare the question is: Rare
Rare: the answer reads well, 10 out of 10, and is true about 65 times in 100. The pile is thin here. The guesses still come out in full, fluent sentences.
Reads well, out of 1010
Is true, in 10040
How rare the question is: Very rare
Very rare: the answer reads well, 10 out of 10, and is true about 40 times in 100. The pile is thin here. The guesses still come out in full, fluent sentences.
Reads well, out of 1010
Is true, in 10010
How rare the question is: Nobody ever wrote it
Nobody ever wrote it: the answer reads well, 10 out of 10, and is true about 10 times in 100. The pile is thin here. The guesses still come out in full, fluent sentences.
The question moves from everyday to something nobody ever wrote about. What changes, and what does not?
Truth falls to almost nothing. The writing reads just as well all the way down.
Right. The model was trained to make text that reads well, and it does that in every frame. Truth was never the thing it was trained on; it came along for free where the pile was thick.
Both fall. A model writes badly about things it does not know.
Not yet. Reads well stayed at 10 in every frame. The model has no sense of not knowing, so its prose does not stumble when its facts run out.
The model says it does not know.
Not yet. Sometimes it does. Mostly it does not, because it has no list to check against and so no moment of noticing. The default is a fluent sentence.
Move the control.
The frame for a question nobody ever wrote about. Tap the bar that stayed the same in every frame.
Tap the part of the picture that answers it.
Reads well, out of 1010
Is true, in 10010
Reads well
Right. Ten out of ten in every frame. Reading well is what the model was trained for, and it delivers it whether or not the facts are there.
Is true
Not yet. This bar fell from 95 to 10 across the frames. It is the one that moved, because truth depends on the pile of text and the pile ran thin.
Are these real?
↓→
Yes, they are
↓→
The database says no
The lawyer asked the chatbot whether the cases were real. It said they were, and could be found in legal databases. That was also text shaped like an answer. Asking a model if it is sure gets you a sentence about being sure. The check has to happen outside the model. The judge fined the lawyers 5,000 dollars.
Each is something a chatbot wrote. Is it the kind of thing it usually gets right, or the kind it often invents?
One card at a time. Tap the pile it belongs to.
Card 1 of 6
The capital of France.
The exact page number of a quotation in a 1987 book.
A short explanation of how rain forms.
The name and date of a court case about your exact situation.
The web address of a small local shop.
A summary of a famous novel.
The capital of France. → Usually right
Not yet. Usually right. The pile of text saying Paris is enormous, so the next-word guess lands on it nearly every time.
The exact page number of a quotation in a 1987 book. → Often invented
Not yet. Often invented. A page number is one small fact in one book, and the pile of text that mentions it is close to nothing.
A short explanation of how rain forms. → Usually right
Not yet. Usually right. Millions of pages explain rain the same way, and the model's guess follows them.
The name and date of a court case about your exact situation. → Often invented
Not yet. Often invented. A case fitting your exact situation may not exist, and the model has the shape of a case ready regardless.
The web address of a small local shop. → Often invented
Not yet. Often invented. An address is exact and rare. The model knows what an address looks like and fills one in.
A summary of a famous novel. → Usually right
Not yet. Usually right. A famous novel has been summarised thousands of times, and the model has seen all of them.
3Usually right
vsvs
3Often invented
Right. Three were common, and the pile was thick. Three were exact and rare, and the pile was thin. The model writes all six in the same confident voice.
A booksomeone wrote it
++
A searcha page exists
++
A chatbottext that reads well
This is a new kind of mistake. A book can be wrong, but somebody wrote it. A search engine shows you a page that exists. A chatbot gives you a sentence built to read well, and it reads just as well when it is false. The feeling that it sounds right is what the model was trained to make.
The studentasks for three studies
↓→
Three studiesauthors and years
A student asks a chatbot for three studies showing that music helps revision, and it gives three, with authors and years. What does the student know at this point?
That three things shaped like studies were written. Whether any exists is not yet known.
Right. Authors and years are part of the shape of a study, and the model has the shape. Existence is checked in a library catalogue, not in the chatbot's answer.
That three studies exist, because they have authors and years.
Not yet. The six court cases had names, dates and quotations. Detail is part of what reads well, and the model supplies it whether or not anything is behind it.
That music helps revision.
Not yet. Even if the three studies exist, the student has not read them. And at this point the student does not know that they exist.
Lesson complete
A chatbot makes text shaped like an answer. The shape is the same whether the answer is true or not.