Lesson 07 · 4 min · 6 things to do
Why the advice contradicts itself
Explain how honest studies produce opposite headlines.
One month a study says a habit helps. The next month another says it does not. Both are honest. What is the most likely difference?
- Yes."Does X help?" is not one question. Helps whom, compared with what, measured how, over how long — change any of those and the honest answer changes with it.
- Not quite.Funding matters and is checkable. It is a much smaller source of contradiction than differences in design.
- Not quite.Two studies of different questions giving different answers is science working. It looks like unreliability only when the headline drops the design.
A study answers the exact question it asked. Headlines report a broader question, and the gap between the two is where the contradictions live.
Which of these differences could flip a result on its own?
Studying trained athletes rather than sedentary adults.
Following people for six weeks rather than three years.
Comparing against nothing, rather than against a real alternative.
Running the study in a different country.
Publishing in a different journal.
Yes.The first three change what was actually tested. The last two change where and by whom — worth knowing, and not able to reverse a finding by themselves.A real effect helps some people and not others. Slide to change who was in the study.
Mostly people it helpsA mixed groupMostly people it does not- Helped70% of participants
- No effect25% of participants
- Worse5% of participants
Mostly people it helpsA clear positive result. True, and about this group.
- Helped33% of participants
- No effect40% of participants
- Worse27% of participants
A mixed groupThe average lands near zero. "No significant effect" — while a third of them were helped.
- Helped10% of participants
- No effect55% of participants
- Worse35% of participants
Mostly people it does notA clear negative result. Also true, also about this group.
The habit did exactly the same thing to each individual in all three frames. What produced three different headlines?
- Yes.An average is a summary of a group, and it can describe nobody in it. That is the single most useful thing to know about any health finding reported as one number.
- Not quite.All three are run correctly. They sampled differently, which is a design difference rather than an error.
- Not quite.Chance widens the spread around each result. It cannot produce three consistent, opposite findings from the same underlying effect.
Move the control to see what changes.
A finding is reported as "a 40% increase in risk". What is missing?
- Yes.40% more of a large risk is a serious matter; 40% more of a one-in-a-million risk is not. The starting number is what makes the increase mean anything, and it is the number most often left out.
- Not quite.Important for judging confidence. It cannot tell you the size of the thing being increased.
- Not quite.Worth knowing and unrelated to what the percentage means.
A risk rises from 2 in 10,000 to 2.8 in 10,000. That is a 40% increase. How many extra cases is that per 10,000 people?
casesYes.Under one extra case in ten thousand. The 40% and the 0.8 describe the same finding, and they lead a reader to completely different conclusions — which is why the choice of which to print is an editorial decision, not a neutral one.You have finished this course. What is the honest thing to say about how your own body handles a hard year?
- Yes.The mechanism is worth knowing precisely because it is general: mobilise, postpone maintenance, settle, repair. What any particular person should do with it is not something a course can know.
- Not quite.Every lesson here has pointed the other way — the same load lands differently depending on what is already on the pile.
- Not quite.The mechanism applies to everyone. It is the numbers and the thresholds that are personal.