Picture a game. I have a secret rule for sets of three numbers, and 2, 4, 6 fits it. You can try any three numbers and I will say yes or no. Most people try something like 8, 10, 12. Yes. Then another set going up by two. Yes. Then they announce: goes up by two. Wrong.
8, 10, 12could only say yes
vsvs
1, 2, 3could have said no
The rule was any three numbers going up. Every test the players tried was one their own guess would pass, so every answer was yes, and every yes felt like proof. A test that can only say yes tells you nothing. The useful test is the one that could have said no: 1, 2, 3. Or 6, 4, 2.
1, 2, 3
++
14, 16, 18
++
100, 102, 104
Your guess is goes up by two. Which of these tests could prove the guess wrong?
100, 102, 104. Big numbers test a rule harder.
Not yet. The size of the numbers does not matter; what matters is whether the guess could fail. The guess says yes to this, and so does the real rule.
14, 16, 18. It is a fair test of the rule.
Not yet. It is a test the guess passes. A yes was certain before you asked, so the yes carries no information; it is 8, 10, 12 again with bigger numbers.
1, 2, 3. If the answer is yes, the guess is wrong.
Right. Goes up by two says 1, 2, 3 is a no. If I say yes, the guess is dead. The other two tests would get a yes from the guess and from the real rule alike, and tell you nothing.
In the 1960 study that invented this game, 29 students played it. How many announced the right rule on their first go?
Pick one. Nothing is scored.
6 of 29
right on their first announcement, out of 29
A few.Six of the 29. Twenty-two of the others announced a wrong rule first, after a run of yeses, and one never announced a rule at all. Almost every test they tried was one their guess would pass. Guesses land at about half, because the game looks easy, and it is; the trap is in which tests feel worth trying.
Move the control through the tests, one at a time. The bar counts how many of four possible rules still fit everything that has been said.
Goes up by 21 still fits: 1, ruled out: 0
Even, going up1 still fits: 1, ruled out: 0
Any three going up1 still fits: 1, ruled out: 0
Any three numbers1 still fits: 1, ruled out: 0
Tests tried: No test yet
Four rules could explain 2, 4, 6. Nothing has been tried.
Goes up by 21 still fits: 1, ruled out: 0
Even, going up1 still fits: 1, ruled out: 0
Any three going up1 still fits: 1, ruled out: 0
Any three numbers1 still fits: 1, ruled out: 0
Tests tried: 8, 10, 12: yes
Yes. All four rules said it would be a yes. Nothing has been ruled out; the test could only agree.
Goes up by 21 still fits: 1, ruled out: 0
Even, going up1 still fits: 1, ruled out: 0
Any three going up1 still fits: 1, ruled out: 0
Any three numbers1 still fits: 1, ruled out: 0
Tests tried: 20, 22, 24: yes
Yes again. All four rules still fit. Two tests, and the count of possible rules has not moved.
Goes up by 20 still fits: 1, ruled out: 0
Even, going up0 still fits: 1, ruled out: 0
Any three going up1 still fits: 1, ruled out: 0
Any three numbers1 still fits: 1, ruled out: 0
Tests tried: 1, 2, 3: yes
Yes. Goes up by 2 and even, going up both said this would be a no. They are gone. Two rules left.
Goes up by 20 still fits: 1, ruled out: 0
Even, going up0 still fits: 1, ruled out: 0
Any three going up1 still fits: 1, ruled out: 0
Any three numbers0 still fits: 1, ruled out: 0
Tests tried: 3, 2, 1: no
No. Any three numbers said this would be a yes, and it was not. One rule left: any three going up.
Which single test did the most work?
3, 2, 1. It was the only one that got a no.
Not yet. The no mattered: it ruled out any three numbers. But that was one rule; the third test ruled out two. A no is not worth more than a yes; what counts is how many rules a result kills.
1, 2, 3. It knocked out two rules at once.
Right. The first two tests ruled out nothing, because every rule said yes to them. The third was the first one that could have said no under two of the rules, and it did.
8, 10, 12. It was the first test, so it set everything up.
Not yet. It ruled out nothing. All four rules said it would be a yes, so the yes could not tell them apart. A test every rule passes is no test.
Move the control.
A horoscopefits any month
vsvs
Rain tomorrow, 80 in 100could be wrong by lunch
The same thing is true of a claim. A claim that nothing could prove wrong says nothing. You will face a challenge this month fits every month that has ever happened. Rain tomorrow, 80 in 100 could be wrong by lunchtime. The second claim says something because it could fail.
Each is a claim. Could anything that happened show it false, or does it fit whatever happens?
One card at a time. Tap the pile it belongs to.
Card 1 of 6
The 8:10 bus will arrive before 8:20.
You will face a challenge this month.
This tablet brings a fever down within an hour.
Everything happens for a reason.
This coin is fair: about half heads over a thousand flips.
If the cure did not work, you did not believe in it enough.
The 8:10 bus will arrive before 8:20. → Could be proved wrong
Not yet. Could be proved wrong. Stand at the stop with a watch; if the bus comes at 8:25, the claim is dead.
You will face a challenge this month. → Fits whatever happens
Not yet. Fits whatever happens. Every month holds something somebody could call a challenge, so no month could count against it.
This tablet brings a fever down within an hour. → Could be proved wrong
Not yet. Could be proved wrong. Give it to a hundred feverish people with a thermometer, and the hour either passes with the fever down or it does not.
Everything happens for a reason. → Fits whatever happens
Not yet. Fits whatever happens. Whatever happens, a reason can be named afterwards, so nothing that could happen would ever count against it.
This coin is fair: about half heads over a thousand flips. → Could be proved wrong
Not yet. Could be proved wrong. Flip it a thousand times; seven hundred heads would sink the claim, lesson 5.
If the cure did not work, you did not believe in it enough. → Fits whatever happens
Not yet. Fits whatever happens. Every failure is explained by the same excuse, so the cure can never lose; which means the claim never said anything.
3Could be proved wrong
vsvs
3Fits whatever happens
Right. Three claims name a result that would sink them, and three have an answer ready for every result. Only the first three are saying anything about the world.
Say the claim→Say what result would sink it→Then look
The word for a claim that could be proved wrong is testable. A testable claim names what would count against it. Scientists write that down before they look: if the new drug does no better than a sugar pill in a hundred people, the idea is wrong. Then they look.
Lost, socks on
↓→
Would have lost by more
A friend says their lucky socks make their team win. The team loses while the socks are on. The friend says: they would have lost by more without the socks. What has just happened to the claim?
It was proved wrong. The team lost with the socks on.
Not yet. It would have been, if the claim had stayed where it was. The friend moved it, and the moved claim cannot be proved wrong by anything, which is the real problem with it.
It was proved right. The team might really have lost by more.
Not yet. Might have is not a result; it is the excuse that arrives after every loss. Nothing that could happen on the pitch would make the friend give the socks up.
It moved so that no result could count against it. It now says nothing.
Right. A win proves the socks and a loss proves the socks. A claim that both results support has stopped being a claim about the world; it fits whatever happens.
Lesson complete
A test that can only say yes tells you nothing. A claim that fits whatever happens says nothing.