Lesson 05 · 4 min · 6 things to do
Why it invents things
Explain confident nonsense as a property of prediction, not a bug.
You ask for three papers on a niche subject. The model gives three, with authors, journals and years. Two do not exist. What went wrong?
- Yes.Citations have a strong shape: name, year, plausible title, plausible journal. Filling that shape is the same operation as any other prediction, and nothing in the model checks whether the result exists.
- Not quite.There is no separate step where it decides to invent. The invented citation is produced by exactly the same process as a correct one.
- Not quite.With no tool attached there is no search. Everything came out of the pattern.
It cannot tell you it does not know, because it does not have a record of what it saw. It has weights, and weights always produce an answer.
The same question, asked about subjects with different amounts of material behind them.
A great dealSomeAlmost none19 of 20 · answers that hold up when checked
A great dealThe pattern is strongly constrained. The answer is usually right, and reads confidently.
14 of 20 · answers that hold up when checked
SomeMostly right, with details drifting — a date out by a year, a name attached to the wrong work.
4 of 20 · answers that hold up when checked
Almost noneFully formed, fluent, and largely invented. It reads exactly like the first frame.
What is the same across all three frames?
- Yes.This is the single most important thing to carry out of this course. The reader's usual guide to how much to trust a claim is disconnected from whether it is true.
- Not quite.The accuracy is what changes. It is the one thing you cannot see.
- Not quite.Answers on thin ground are often LONGER, because there is less to pin them down.
Move the control to see what changes.
Which of these answers can be checked cheaply, and which would cost real work?
A named paper with a title and year.
A quoted line of law with a section number.
A general claim about "most studies".
A confident summary of an unnamed report.
A calculation you can redo yourself.
Yes.Specific answers are falsifiable and vague ones are not — so the answer that sounds safest is often the one you can never test. Ask for the specific version.You correct the model. It apologises and produces a different answer, just as confidently. What have you learned?
- Yes.It will also "correct" a right answer if you push. Agreement under pressure is a fact about the conversation, not about the world.
- Not quite.The second answer is produced the same way as the first, with a strong pull toward agreeing with you.
- Not quite.There is nothing to withhold. Both answers were generated on the spot.
You ask for 10 references. Each has an independent 15% chance of being invented. Roughly how many would you expect to be fake?
referencesYes.One or two, every time — and they are mixed in with eight real ones, which is what makes the list so convincing. A list is not safer than a single claim; it has more chances to be wrong.Which working habit reduces the damage most?
- Yes.It matches the tool to what it is good at. Structure, drafting, rewording and explanation are where the prediction is strong; a particular date is where it is weakest.
- Not quite.Safe and pointless. The value is in the work you cannot already do.
- Not quite.Bigger models invent less on thin ground. They still do it, in the same voice — which is the next lesson.
Lesson complete
Invention and correct answers come out of the same machinery, so confidence tells you nothing.
Next: What bigger buys →