Picture a test for a rare illness. One person in a thousand has it. The test is 99 per cent accurate: it catches 99 in 100 who have it, and it wrongly flags 1 in 100 who do not. Somebody gets a positive result. Most people, doctors included, think they are 99 per cent likely to be ill.
Flagged, and ill1 people
Flagged, and well10 people
Out of 1,000 tested.
Count it. Of the thousand, one is ill, and the test catches them. The other 999 are well, and the test wrongly flags 1 in 100 of them: about 10 people. So about 11 people are flagged. One of them is ill. Ten are well and frightened.
1 of 11
ill, out of eleven flagged
Eleven people are flagged and one is ill. For a person holding a positive result, the chance they are really ill is nearest to which?
About 1 in 11, under 10 in 100.
Right. One ill person among eleven flagged. The test did its job on everybody; there were simply a thousand well people for it to be wrong about, and one ill person for it to be right about.
About 99 in 100. The test is 99 per cent accurate.
Not yet. The 99 belongs to the test, not to the person. The test is right 99 times in 100 on each person, and there are 999 well people for its one-in-a-hundred mistake to land on.
About 1 in 1,000, the same as before the test.
Not yet. The test moved the number. Before it, the chance was one in a thousand; after a positive result it is one in eleven. That is a big move, and still a long way from likely.
1 in 100 flagged: 10999 well
vsvs
99 in 100 flagged: 11 ill
The number everybody skipped is how rare the illness was before anybody tested anything. That number is called the base rate: how common a thing is to start with. When the thing is rare, even a good test flags more healthy people than ill ones, because there are so many more healthy people to flag.
Put the count in order, the way the doctors who got it right did it.
Tap them in the order they happen, first to last. Each tap gets a number.
1,000 people→1 ill, 999 well→1 flag, 10 flags→1 of 11
Right. People first, then the split, then the test, then the comparison. Doing it in this order keeps the 999 well people in view; doing it with percentages lets them vanish behind the 99.
Move the control through how many people in 1,000 have the illness. The test stays the same: it catches 99 in 100 who have it and wrongly flags 1 in 100 who do not. Computed.
Flagged, and ill1 people
Flagged, and well10 people
People in 1,000 who have it: 1
1 in 1,000 have it. The test flags 1 of the people who have it and 10 who do not. A flagged person is really ill about 9 times in 100.
Flagged, and ill9.9 people
Flagged, and well9.9 people
People in 1,000 who have it: 10
10 in 1,000 have it. The test flags 9.9 of the people who have it and 9.9 who do not. A flagged person is really ill about 50 times in 100.
Flagged, and ill99 people
Flagged, and well9 people
People in 1,000 who have it: 100
100 in 1,000 have it. The test flags 99 of the people who have it and 9 who do not. A flagged person is really ill about 92 times in 100.
Flagged, and ill495 people
Flagged, and well5 people
People in 1,000 who have it: 500
500 in 1,000 have it. The test flags 495 of the people who have it and 5 who do not. A flagged person is really ill about 99 times in 100.
At which base rate is a positive result a coin flip, with about half the flagged people really ill?
1 in 1,000. The test is 99 per cent accurate, so it must be close to even.
Not yet. At 1 in 1,000 the ill pile is one person and the well pile is ten. A flagged person is ill about one time in eleven, nowhere near a coin.
500 in 1,000. Half the people have it, so half the flags are right.
Not yet. At 500 in 1,000 the test flags 495 ill people and 5 well ones. A flag is right 99 times in 100, far past a coin flip.
10 in 1,000. The test flags about 10 ill and about 10 well.
Right. At 10 in 1,000 the two piles are the same size, so a flagged person is ill about half the time. Below that the well pile is bigger; above it the ill pile is.
Move the control.
1 thief1,000 shoppers
↓→
about 11 timesThe scanner beeps
The same count runs far from medicine. A shop's security scanner wrongly beeps at 1 in 100 honest shoppers, and 1 shopper in 1,000 is stealing. So on a day with a thousand shoppers it beeps at about ten honest people and one thief. Most people stopped at the door did nothing.
In a 1998 study, 48 doctors were given four rare-illness problems like this one, with the numbers as percentages. Slide the marker to the share of their answers that were right.
Slide the marker to where you think the answer is, then press Lock. Nothing is scored.
123
your guess · 0%
the answer · 10%
0%50%100%
1A quarter25%
2Half50%
3Most90%
Your guess: —
About one answer in ten. Given the same problems as counts, 1,000 people, 10 flags, the share of right answers rose to nearly half. The maths was the same; the counts kept the 999 well people in the room, and the percentages let them disappear.
99% accurate→1,000 people, 1 ill→11 flagged→1 in 11
That is why the count was written out in people, not percentages. Ninety-nine per cent sounds like a fact about the person in the chair. One of eleven is a fact about the room they came from, and the room is where the base rate lives.
2 caught2 spam a day
vsvs
2 flagged200 real a day
A spam filter catches 99 in 100 spam mails and wrongly flags 1 in 100 real ones. You get 2 spam mails a day and 200 real ones. A mail is in the spam folder. How likely is it to be spam?
About 1 in 100. Spam is rare, so a flagged mail is almost never spam.
Not yet. Rare is the base rate, and the filter moved the number from one in a hundred to about one in two. The folder is not mostly spam, and it is not nearly empty of it either.
About 99 in 100. The filter is 99 per cent accurate.
Not yet. The 99 is the filter's accuracy on each mail. Two hundred real mails give its one-in-a-hundred mistake two chances a day, and two spam mails give it two hits. The folder is half and half.
About half. Two spam caught, two real ones flagged.
Right. The filter is as good as the illness test, and the base rate is one spam in a hundred mails. Two caught against two wrongly flagged makes the spam folder a coin flip.
Lesson complete
How rare a thing is comes first. For a rare illness, a good test flags more well people than ill.