Picture a headline: green jelly beans linked to acne. Behind it, a lab tested twenty colours of jelly bean. Nineteen showed nothing. The green one showed a link. The green result went to the newspaper and the other nineteen went into a drawer.
1 of 20
test out of twenty that passes by chance alone
Twenty harmless tests at a 1-in-20 bar.
Scientists have a rule for calling a result real: chance alone would produce it fewer than one time in twenty. That is the bar. So if you test twenty harmless things, about one of them will clear the bar by chance. That one gets the headline, because it is the one that looks like something.
1 of 20
passes by chance, out of twenty
Twenty colours are tested and none of them really does anything. About how many pass the 1-in-20 bar by chance?
About ten. Half of all tests are flukes.
Not yet. One in twenty is one in twenty, not one in two. Ten flukes would need a bar set at a coin flip, and no lab sets it there.
About one.
Right. One in twenty is the bar, and there are twenty tries. On average one harmless colour clears it, and the lab cannot tell from the result alone which kind of pass it was.
None. Harmless things do not pass a test.
Not yet. The bar is set so that a harmless thing passes one time in twenty by chance. It is a bar, not a wall, and twenty tries give chance twenty goes at it.
Move the control through the number of harmless things a lab tests. The bar is 1 in 20 each. The chance that at least one passes is computed.
At least one passes by chance5 in 100
None passes95 in 100
Harmless things tested: 1
One harmless test. It passes the 1-in-20 bar by chance about 5 times in 100.
At least one passes by chance23 in 100
None passes77 in 100
Harmless things tested: 5
5 harmless tests. The chance that at least one passes the 1-in-20 bar by chance alone is about 23 in 100.
At least one passes by chance64 in 100
None passes36 in 100
Harmless things tested: 20
20 harmless tests. The chance that at least one passes the 1-in-20 bar by chance alone is about 64 in 100.
At least one passes by chance99 in 100
None passes1 in 100
Harmless things tested: 100
100 harmless tests. The chance that at least one passes the 1-in-20 bar by chance alone is about 99 in 100.
A lab runs a hundred tests on things that do nothing. What is the chance that at least one passes the bar?
About half.
Not yet. Half is roughly where fourteen tests land. By a hundred, chance has had so many goes that a pass is all but certain.
About 5 in 100. The bar is 1 in 20.
Not yet. One in twenty is the chance per test. A hundred tests give chance a hundred goes, and the chance of at least one pass climbs to about 99.
Nearly certain, about 99 in 100.
Right. A hundred tries at one in twenty each almost guarantees a pass, and the lab will have about five of them. Each one looks exactly like a real finding.
Move the control.
Tries that failedin the drawer
↓→
The one that passedon the news
The drawer is the problem, not the bar. A result on its own cannot say how many tries came before it. One study found is a sentence about the drawer: it tells you one try passed and says nothing about how many did not. The headline is the survivor from lesson 6.
1,000→500 saw him right→250 saw him right twice→? saw him right three times
Picture a tipster. He posts to 1,000 people: 500 are told shares will rise, 500 that they will fall. Next month he posts only to the 500 he was right with, split again. After three rounds, how many have seen him right three times running?
people
Halve a thousand, three times.
Right. One hundred and twenty-five. Each of them has watched him call it right three months in a row, and each would swear he has a gift. The 875 who saw him wrong are the drawer, and none of them is on his mailing list any more.
Say what you will test, first→Run it→Run it again, elsewhere
Science has two fixes, and both are about counting the drawer. Say in advance what you will test and how, so a failed try cannot quietly vanish. And do the test again, somewhere else, with new people. A result that passes twice, in two places, has beaten chance twice.
Each is a result on show. Does it count every try, or show only the winners?
One card at a time. Tap the pile it belongs to.
Card 1 of 6
A tipster's five best predictions of the year.
A clinic's success rate across every patient it treated.
The testimonials on a product's web page.
A trial that said in advance what it would measure, then reported it.
A psychic's famous hits.
A school's exam results for all 200 pupils who sat the paper.
A tipster's five best predictions of the year. → Shows the winners
Not yet. Shows the winners. Five best means there were others, and best is the gate those others did not come through.
A clinic's success rate across every patient it treated. → Counts every try
Not yet. Counts every try. Every patient is in the count, including the ones for whom it did not work.
The testimonials on a product's web page. → Shows the winners
Not yet. Shows the winners. A testimonial is written by a buyer it worked for, and chosen by a seller; the buyers it failed wrote nothing.
A trial that said in advance what it would measure, then reported it. → Counts every try
Not yet. Counts every try. Saying it in advance means the drawer was locked before the result arrived.
A psychic's famous hits. → Shows the winners
Not yet. Shows the winners. A psychic makes hundreds of guesses, and the misses are forgotten by the people who heard them.
A school's exam results for all 200 pupils who sat the paper. → Counts every try
Not yet. Counts every try. All 200 means nobody who sat the paper was left out of the average.
3Counts every try
vsvs
3Shows the winners
Right. Three results carry their failures with them, and three have left the failures somewhere you cannot see. Only the first three can be read as evidence for anything.
Right eight years running
↓→
First question?
An advert: our stock tips beat the market eight years running. By this lesson, what question comes first?
How many tipsters started eight years ago, and where are the ones who lost?
Right. Eight years running is the winner's record. If a thousand tipsters started, chance alone leaves a few with a perfect run, and the advert is written by one of them.
How much the tips cost.
Not yet. The price tells you what the advert wants. The record tells you nothing until you know how many records were made and how many were thrown away.
Whether eight years is long enough to prove it.
Not yet. Eight years would be plenty, if it were the only try. The question is not the length of the run but the size of the drawer it came out of.
Lesson complete
Run enough tries and one passes by chance. The question for any result is where the tries that failed went.