Lesson 06 · 4 min · 6 things to do
The trials you never see
Predict how missing results distort a published average.
Ten trials are run on a drug that does nothing. Two happen, by chance, to look positive. Which two get published first and most prominently?
- Yes.Nobody has to act dishonestly. Researchers write up exciting findings first, journals prefer them, and sponsors have no reason to push a disappointing result into print.
- Not quite.A large share of completed trials are never published at all, and negative ones are much the most likely to be missing.
- Not quite.Negative results are the hardest to place and the least cited.
The published literature is a sample of the research done, and it is not a random sample. Interesting results are more likely to reach print.
Twenty trials are run on a drug with no real effect. Slide to change how many get published.
All 20Positives plus a fewPositives only- Positive by chance2 trials · 10%
- Null, and published18 trials · 90%
All 20Eighteen null, two positive by chance. The average across them is correctly about zero.
- Positive by chance2 trials · 29%
- Null, and published5 trials · 71%
Positives plus a fewTwo positive and five null in print. The average now looks mildly favourable.
Positive by chance2 trialsNull, and published0 trialsPositives onlyTwo positive trials in the literature and nothing else. The drug appears to work, twice.
In the last frame, what would a careful reviewer who reads everything published conclude?
- Yes.This is why reviews search trial registries for studies that were started and never reported. Reading everything published is not the same as reading everything found.
- Not quite.Two positive trials with no contradicting evidence is, on its face, a reasonable case. That is what makes the distortion so effective.
- Not quite.There is nothing visible to be suspicious of. The missing trials leave no trace in the literature — only in the registry.
Move the control to see what changes.
Which of these help detect missing results?
Requiring every trial to be registered before it starts.
Comparing published outcomes against the registered plan.
Checking whether small trials report unusually large effects.
Reading more published papers.
Trusting well-known journals.
Yes.The missing results are, by definition, not in the literature. Only a record made BEFORE the results existed can show you what should have been there.A trial was registered to measure deaths, and the published paper reports a symptom score instead. What most likely happened?
- Yes.It is called outcome switching, and comparing papers against their registrations finds it constantly. The registry is what makes it visible at all.
- Not quite.Sometimes stated, and it is checkable — the paper can say so and report it later.
- Not quite.It may well be, and the time to argue that was before the results were known.
Twenty trials of a useless drug are run at the usual 5% threshold. Roughly how many would you expect to look positive by chance alone?
trialsYes.One in twenty, which is what that threshold means. Publish only that one and the drug has a published evidence base — built entirely out of the noise the threshold was designed to allow.What does this mean for reading a single impressive study?
- Yes.The question 'what else was tried?' is often unanswerable from the paper, and it is the one that decides how much weight the finding deserves.
- Not quite.A well-run large trial can be decisive. The caution is about the small, striking, isolated result.
- Not quite.Reviews help, and they inherit the same distortion unless they go to the registries.
Lesson complete
The published literature is a sample of the research done, and not a random one.
Next: Approval is about a population →