Confidence and the black box

A classifier looks at a photo and prints one line: cat 91%. That number feels like a promise. Ninety-one out of a hundred — surely it’s almost certainly a cat, and surely the machine has some reason for saying so. Both feelings are wrong. The 91% is not a probability of truth, and there is no reason underneath it that anyone can put into a sentence. Two things worth understanding sit inside that one small number.

What a confidence number actually is

When the model shows “cat 91%”, it is not reporting the odds that the animal is really a cat. It is reporting how strongly the input lit up the cat pattern it built during training, compared to every other pattern it knows. It’s a match score, not a measurement of reality.

Here’s a detail that gives the game away. The scores across the options are forced to add up to 100%. Cat, dog, fox — the machine spreads a fixed hundred points across whatever labels it was trained on, and hands the most to the closest match. So it always names a most-likely option. Show it a photo of a shoe and it will still answer with a cat-or-dog-or-fox split summing to 100, because it has no box for “none of these” and no way to say “I’ve never seen anything like this.” The number is a ranking of its own patterns, dressed up to look like certainty.

And because the score is only a pattern match, it carries none of the caution a person would. Back in the item on confident errors, we saw fluency and truth come apart. This is the same split, wearing a percentage. A high score can sit squarely on a wrong answer. Ninety-one percent means strongly matches what I saw before — nothing more.

Why the number is brittle

If the score is a pattern match, then it moves whenever the input drifts away from the patterns the model was trained on. And it doesn’t drift gently. Blur the photo. Cover part of it. Sprinkle in specks of noise a person would barely register. Small nudges like these can swing the score hard, because the machine was never shown that exact roughed-up version and has no steady sense of the animal underneath — only the surface pattern, which you just disturbed.

A worked example. An image classifier is shown a clear photo of a cat. It outputs:

• cat 91% · dog 6% · fox 3% (they sum to 100)

It looks certain. Now add a light blur and a small patch over one ear — a change most people would shrug at. The very same model now outputs:

• dog 74% · cat 21% · fox 5%

Nothing about the animal changed. A tiny push outside its training patterns flipped a confident “91% cat” into a confident “74% dog.” The number stayed high; the answer became wrong. That is the whole warning: read confidence as “how strongly this matches what I saw before,” not “how sure I am of the truth.” A brittle score can be loud and still be lost.

The black box: no reason underneath

Now ask the obvious question. Why did you decide dog? And there is no plain answer to give.

The decision is the end of an enormous chain of tiny multiplications and additions — millions of tuned numbers, passed through layer after layer, as we saw when we built up the training loop and the layered net. No single number in there is “the reason.” There is no rule, no clause, no because. The output is simply what all those weighted sums happened to total on this particular input.

This is why people call it a black box. You can see the input. You can see the output. The middle is opaque — not because it’s hidden from you on purpose, but because there is no readable sentence in there to hide. Even the people who built it can open up the weights, stare at every one, and still not tell you, in words, why this photo got this answer. There are tools that can hint at which parts of an input mattered most — which pixels tugged the score toward “dog.” But a hint about what mattered is not the same as a clean reason you could read out and check. Usually, that reason simply isn’t available.

Why this matters

Put the two together and you have the shape of the risk. A confidence number is a brittle pattern-match score, not a promise of truth. And the decision behind it comes with no explanation anyone can give. For a low-stakes guess — sorting holiday snaps, suggesting the next word — that’s fine. Nobody gets hurt if “cat 91%” was really a small dog.

But the danger arrives when a confident score with no readable reason is handed to a decision that people take on trust: who gets the loan, which scan gets a closer look, whether a car brakes. There the missing “because” stops being a curiosity and becomes the whole problem. You can’t argue with a number that won’t explain itself, and you can’t easily catch it being confidently, brittlely wrong.

On the whole, the honest way to read a machine’s confidence is to shrink it back to what it is: a score saying this input resembles things it saw before, printed by a system that cannot tell you why. That’s a useful thing and a limited thing at the same time. Knowing the difference — trusting the loud number a little less, and asking for the reason a little more — is most of what it takes to use these machines without being fooled by them.