The morning a “positive” trial crashed

MapLight’s schizophrenia drug did what a drug is supposed to do. It hit its main goal. The result was statistically significant, the phrase every press release wants. Then the stock fell more than 60% before lunch.

Nothing had gone wrong with the trial. The drug really did beat the placebo, and the odds that this was a fluke were low. The market simply asked a different question than the headline answered. The headline said the effect was real. The market asked how big it was. Those are two separate questions, and the whole confusion of the week lives in the gap between them.

What “significant” actually promises

Here is the thing almost no one is told: “statistically significant” is a yes-or-no answer to one narrow question. Is this effect probably real, or could it just be the random noise of a small sample? That is all it certifies. It is a fluke-detector, not a size-meter.

Whether an effect counts as significant depends on two things: how big the effect is, and how many people you tested. Test enough people and even a microscopic effect becomes “significant.” Not because it grew, but because a bigger sample makes it easier to tell a small real thing apart from noise. Significance is partly a statement about the drug and partly a statement about the size of your study.

So a giant trial can prove, beyond reasonable doubt, that a drug does almost nothing. The effect is real. The effect is also useless. Both are true at once, and only one of them shows up in the word “significant.”

The size is a different number entirely

The question that actually matters to a patient is not “is there an effect” but “is the effect large enough to change my life.” That is the effect size, and it is a completely separate measurement from significance.

MapLight’s drug lowered symptom scores by 4.5 points more than placebo. Real, yes. But smaller than the approved rival already on pharmacy shelves. Investors were not doubting the result was genuine. They were measuring its size against what a patient would need, and finding it short. The p-value said “this happened.” The 4.5 said “not by enough.”

You can see the two numbers move independently if you watch for them. A trial can be significant and trivial. It can be non-significant and, if only it were bigger, genuinely important - MapLight’s once-daily dose pointed the right way but was too small a study to prove it. The press release reaches for the same warm word in every case: positive.

The mirror trap: a real cut in a risk that was already tiny

The gap runs the other way too, and Lilly’s obesity drug walked straight into it. The weight loss was enormous, over 22%. But the trial also asked whether the drug prevents heart attacks and strokes, and there the answer went blurry: 44 events in the treated group, 52 in the placebo group.

Why so murky, when the weight numbers were so clean? Because heart attacks were rare in both groups to begin with. You cannot measure a difference in something that barely happened. When a risk is already near the floor, even a drug that genuinely helps has almost no room to show it. The trial goes quiet, not because the drug failed, but because there was nothing large to detect.

This is where a true result gets oversold. “Cuts the risk by 40%” is how the long-COVID drug news arrived this week, and it may well be a real finding. But 40% of a small risk is a small number. Cutting a 2-in-100 risk to a bit over 1-in-100 is a 40% cut and a tiny absolute change at the same time. The relative number sounds like a breakthrough. The absolute number tells you what actually changed for a real person. A headline almost always quotes the first and drops the second.

Who reads the word, and what they hear

Watch how far the single word “positive” travels, and who it touches. The company writes it because it is technically true. The wire picks it up. A patient’s family reads “hits its main goal” and hears “this works.” A doctor scanning the abstract files it as a win. Each person is downstream of a word that certified the effect exists and quietly said nothing about whether it matters.

And notice the structure underneath. Someone chose the bar. Someone decided which outcome counted as the “primary endpoint,” what threshold marked success, how many patients to enrol. Those choices decide whether a trial gets to say “positive” at all. And they are choices, made by people with something riding on the answer, not facts handed down by nature. A regulator and a company can stare at the identical Duchenne dataset and disagree on whether there was any effect, because the dataset alone never settles it. Judgement does.

None of this makes the science dishonest. The trials were run properly; the results are what they are. It makes the reader’s job larger than reading the verdict. Two questions hide inside every “positive”: is it real, and is it big. The first is the one the statistics answer. The second is the one you have to ask yourself. The market, the patient, and the regulator this week each answered it differently, from the same set of numbers, none of them seeing the whole.