Capstone: reading an AI claim

You made it here for a reason

Every item in this course pried open one part of the machine. Now they come together into a single skill: reading an AI claim and knowing, roughly, how much to believe it. Not with a lab tool this time — in your own head, the moment a claim scrolls past. The claims are everywhere. A product says its model understands you. A report says a system is 97% accurate. An article says an assistant made something up, so it must be broken. Each one is doing a specific thing you now have the eyes to see.

The trick is that most claims aren’t a clean lie or a clean truth. They’re a mix — a real grain, wrapped in more certainty than it earns. The skill is separating the grain from the wrapping.

A short toolkit of questions

Six questions, drawn straight from the course. You won’t need all six on every claim; one or two usually crack it open.

What is it really doing? An AI predicts patterns; it does not know things. When a claim says a system understands, thinks, or knows, hold it up to that light. Is real comprehension being described, or is fluent prediction being dressed up as understanding?

Who was its teacher? The data taught it, and the data has holes. Where the examples were thin, the machine is blind. A claim that a model works well “in general” is only as good as the examples behind it — and your case might sit in one of the gaps.

Is a score being sold as certainty? A high accuracy number or a confidence percentage is a pattern-match strength, not a promise of truth. An average of 97% still hides individual mistakes made with full confidence. Ask whether a number that measures pattern strength is being passed off as reliability for you.

Could it be confidently wrong? A prediction machine will fill a gap with something fluent and plausible whether or not it’s real. It has no sense of true or false, only likely. So “it sounded sure” tells you nothing about whether it was right.

Is bias just being hidden? Removing a sensitive field from the data doesn’t remove the pattern — the model rebuilds the skew through other clues that stand in for it. A claim that a system “can’t be biased now” because one column was deleted has usually just hidden the bias, not removed it.

Does this task even suit a machine? Pattern-rich and error-tolerant is a good fit. Novel, high-stakes, and needing someone accountable is a job for a human. When a claim puts a machine in charge of the second kind, that’s the flag.

Three verdicts

Once you’ve run the questions, sort the claim into one of three piles.

Sound. True and well-supported. It describes what the machine actually does, claims no more than it can, and doesn’t hide the catch. A claim like “the filter learned the pattern from messages people already marked as spam” is just an accurate account of learning from examples. Nothing to argue with.

Shaky. There’s a real grain, but it’s overstated or stripped of the context that would change your mind. The number is real but oversold; the finding is real but the caveat is missing. These are the slippery ones, because the grain is genuine — you have to catch the gap between what’s true and what’s implied.

Oversold. Wrong, or misleading enough that acting on it would hurt you. It claims understanding where there’s only prediction, or certainty where there’s only a hidden guess. Not always a lie — often just fluency mistaken for something deeper.

Decoding one claim, all the way through

Take this one: “This AI genuinely understands the questions you ask it, the way a person does.”

Run the first question — what is it really doing? From the whole course, the answer is settled: it predicts a likely continuation from patterns in its training data. When you type a question, it isn’t grasping your meaning; it’s computing which words tend to follow words like yours, and producing those. That’s a real and useful trick. But there is no comprehension behind it — no picture of your situation, no sense of what the words point at.

So the claim’s key word — understands, the way a person does — is doing work the machine can’t back up. What’s actually happening is fluency: the output reads so smoothly that it feels like understanding, and the claim quietly swaps the feeling for the fact. That’s the fluency-mistaken-for-understanding trap, and it’s one of the most common ways an AI gets oversold.

Verdict: Oversold. Not because the system is useless — it may answer beautifully — but because “understands like a person” is a false description of what’s going on inside. Notice how little you needed: one question cracked it. That’s usually how it goes.

On the whole

You started this course unable to write the rule for a cat. You end it able to read the machine that learned one. That’s the quiet shift — you’re no longer on the outside of the AI-shaped world, taking its claims at face value, and you’re not on the far side either, dismissing all of it as hype. You’re inside it, with a set of questions that tell truth from wrapping.

Keep the humility that comes with the tools. The machine predicts; it doesn’t know — and knowing that is what lets you decide, case by case, how much of any claim to trust. That was the whole point: to see the machinery clearly enough to make humbler decisions inside it. You can, now.