Picture two weather people. One says rain or no rain, every day. The other says a number: 70 in 100. After a year, which one did better? You can only tell by keeping score, and there is a way to score a 70 that is fair to it on the dry days too.
Said 70, it rained9 penalty
Said 70, dry49 penalty
Said 100, dry100 penalty
Gap squared, scaled to 100.
The score: take the gap between the number you said and what happened, 100 if it happened and 0 if not. Square the gap; smaller is better. Say 70 and it rains: 30 away, penalty 9. Say 70 and it stays dry: 70 away, penalty 49. Say 100 and it stays dry: the worst score there is.
Said 100
↓→
Did not happen
A forecaster says 100 in 100 and it does not happen. What is their penalty?
50. They were half right, because it could have gone either way.
Not yet. The score has no half right. The number said was 100 and the world said 0, so the gap is the whole scale, and so is the penalty.
0. One miss is forgiven.
Not yet. The score forgives nothing; that is why it works. A miss at 100 costs everything, which is exactly why good forecasters almost never say 100.
100. The worst score possible.
Right. A hundred away, squared and scaled, is the top of the penalty scale. Certain and wrong is the most expensive thing a forecaster can say.
Set the number the forecaster said and whether it happened. The penalty is computed: the gap, squared, scaled to 100.
Penalty25
Worst possible100
The number said: 50 · What happened: It happened
You said 50 in 100, and it happened. You were 50 away. Squared and scaled: a penalty of 25. Lower is better.
Penalty25
Worst possible100
The number said: 50 · What happened: It did not
You said 50 in 100, and it did not happen. You were 50 away. Squared and scaled: a penalty of 25. Lower is better.
Penalty9
Worst possible100
The number said: 70 · What happened: It happened
You said 70 in 100, and it happened. You were 30 away. Squared and scaled: a penalty of 9. Lower is better.
Penalty49
Worst possible100
The number said: 70 · What happened: It did not
You said 70 in 100, and it did not happen. You were 70 away. Squared and scaled: a penalty of 49. Lower is better.
Penalty1
Worst possible100
The number said: 90 · What happened: It happened
You said 90 in 100, and it happened. You were 10 away. Squared and scaled: a penalty of 1. Lower is better.
Penalty81
Worst possible100
The number said: 90 · What happened: It did not
You said 90 in 100, and it did not happen. You were 90 away. Squared and scaled: a penalty of 81. Lower is better.
Penalty0
Worst possible100
The number said: 100 · What happened: It happened
You said 100 in 100, and it happened. You were 0 away. Squared and scaled: a penalty of 0. Lower is better.
Penalty100
Worst possible100
The number said: 100 · What happened: It did not
You said 100 in 100, and it did not happen. You were 100 away. Squared and scaled: a penalty of 100. Lower is better.
Which costs more: saying 100 and being wrong once, or saying 50 four times?
The wrong 100. Nothing costs more than certain and wrong.
Not yet. Nothing costs more in one go. Four 50s cost 25 each, whatever happens, and four of them add up to the same 100.
The same. One wrong 100 costs 100; four 50s cost 25 each.
Right. Four coin-flip answers cost exactly one certain miss. Saying 50 is safe and says nothing, and the score charges for that too, just more slowly.
The four 50s. Hedging is punished hardest.
Not yet. Hedging costs 25 a time, and four times is 100: the same as one certain miss, not more. The score punishes both, equally, in the end.
Move both controls.
284 experts28,000 forecasts
↓→
3 to 5 years outabout as good as chance
The score has been kept on real experts. From 1984 to 2004 a researcher collected about 28,000 forecasts from 284 experts in politics and economics. On questions three to five years out, the average expert did about as well as chance. The researcher's own phrase was a dart-throwing chimpanzee.
Hedgehogone big idea
vsvs
Foxmany small facts
The average hid two kinds of expert. He called them hedgehogs and foxes. A hedgehog knows one big idea and fits every fact to it. A fox knows many small things and changes its mind in small steps as facts arrive. The foxes scored better. The hedgehogs scored worse the further ahead they looked.
Put a fox's week in order.
Tap them in the order they happen, first to last. Each tap gets a number.
Say a number→A clue arrives→Move it a little→Score it
Right. Number, clue, move, score. The third step is lesson 13 and the fourth is this one. A hedgehog skips the third, because the big idea already said what would happen, and skips the fourth, because a miss is explained rather than scored.
Volunteersthe best 2 in 100
↓→
Small updates, often
Later, in a four-year forecasting tournament from 2011, the same researcher found the best two in a hundred ordinary volunteers and called them superforecasters. The project's forecasts, everyone's put together, beat a prediction market run inside the intelligence services, whose analysts had secret information. What the best ones did: changed their minds often, and by a little each time.
Each is something a forecaster says. Hedgehog, or fox?
One card at a time. Tap the pile it belongs to.
Card 1 of 6
It is always the government's fault, whatever the story is.
I had it at 70 last week. The new figures move me to 60.
This proves what I have been saying for years.
That depends. Let me check three things first.
I would say 65, not yes.
The facts are wrong. My theory is not.
It is always the government's fault, whatever the story is. → Hedgehog
Not yet. Hedgehog. One idea fitted to every story before the story has been read.
I had it at 70 last week. The new figures move me to 60. → Fox
Not yet. Fox. A number, a clue, and a move by about as much as the clue is worth.
This proves what I have been saying for years. → Hedgehog
Not yet. Hedgehog. A new fact is used to confirm the old idea, not to test it; that is lesson 3's can I believe it.
That depends. Let me check three things first. → Fox
Not yet. Fox. Many small facts, gathered before the number is set.
I would say 65, not yes. → Fox
Not yet. Fox. A number is a thing that can be scored and moved; yes is a thing that can only be defended.
The facts are wrong. My theory is not. → Hedgehog
Not yet. Hedgehog. When the facts and the idea disagree, the facts are sent away; a miss is explained instead of scored.
3Hedgehog
vsvs
3Fox
Right. Three keep one idea and bend the facts to it. Three keep a number and bend it to the facts. Over twenty years and 28,000 forecasts, the second kind scored better.
95: it will flop
↓→
Big opening weekend
A friend was 95 sure a film would flop. On its opening weekend it did very well. By this lesson, what does a fox do with the 95?
Moves it down a long way, and keeps the score. A big opening is a strong clue against a flop.
Right. A strong clue moves even a high number, lesson 13, and the fox writes down that the 95 cost them. The old number is not defended; it is scored and replaced.
Jumps to 0. They were plainly wrong.
Not yet. Zero is as stuck as 100, and a strong opening does not make a flop impossible. The fox moves a long way and leaves room, because the next clue is coming.
Keeps the 95. One weekend proves nothing.
Not yet. One weekend is a clue that shows up far more often for films that do not flop. Keeping the number through it is the hedgehog's move: the facts are wrong, the theory is not.
Lesson complete
A forecast with a number can be scored. The best forecasters change their minds often, and by a little.