Picture being refused a loan. You ask why. The bank says the system scored you 61 and the line was 65. You ask what you did wrong. The bank cannot tell you, and not because it is hiding something. Nobody at the bank knows. The reason is spread across millions of numbers.
A clue→Times a number→Times another→Added up→A score
A model, lesson 1, is a very long list of numbers saying how much each clue counts. A rule a person writes has a reason you can read: over £50, postage free. A model found in examples has no sentence in it anywhere. It has the numbers, and the numbers together give the score. No line says because.
The bank
↓→
Millions of numbers
↓→
No sentence saying why
The bank cannot say why you were refused. What is the most likely reason?
The bank is hiding the reason.
Not yet. Banks have hidden things, but this is a different problem. Even a bank that wanted to tell you could not, because the reason was never written anywhere.
The model has no reason written in it. Only numbers that add up to a score.
Right. A written rule carries its reason in words. A model found in examples carries only how much each clue counts. There is nothing to read out.
You were refused at random.
Not yet. A score of 61 is not random; the same form gets the same score every time. It is fixed, and it is still without a reason anyone can say.
A spam filterthousands of numbers
vsvs
A 2020 writing model175 billion
Here is how many numbers. A spam filter might have a few thousand. A model that writes essays, released in 2020, had 175 billion. Read one a second without stopping and you would finish in about 5,500 years. Nobody has read them. Nobody could. The model works, and no person knows what any one number is for.
Slide to the number of numbers in that 2020 writing model, in billions.
Slide the marker to where you think the answer is, then press Lock. Nothing is scored.
12
your guess · 0 billion
the answer · 175 billion
0 billion250 billion500 billion
1One hundred100 billion
2Four hundred400 billion
Your guess: —
One hundred and seventy-five billion, each one set by the examples and none of them written by a person. Guesses tend to land low, because a list of numbers sounds like something somebody could check. This one is longer than every book ever printed.
Change one thing→Watch the score→Guess what mattered
People have tried to get reasons out anyway. One way: change one thing about you and see if the score moves. Change your postcode and the score goes to 70. So the postcode mattered. That is a reason found from outside, by poking. It is a guess about the model, not a line read from it.
Change one thing on the loan form and watch the score. The figures are made up and round. The line is at 65.
Your score61
The line65
What you change on the form: Nothing
Change nothing: the score is 61, below the line of 65. Refused.
Your score62
The line65
What you change on the form: Age
Change the age: the score moves to 62, up 1. Still under the line. You have learned what moves the score. Not why.
Your score66
The line65
What you change on the form: Income
Change the income: the score moves to 66, up 5. That clears the line. You have learned what moves the score. Not why.
Your score58
The line65
What you change on the form: Years at address
Change the years at address: the score moves to 58, down 3. Still under the line. You have learned what moves the score. Not why.
Your score70
The line65
What you change on the form: Postcode
Change the postcode: the score moves to 70, up 9. That clears the line. You have learned what moves the score. Not why.
Poking the form shows that postcode moves the score most. Does that tell you why the model works that way?
Yes. Postcode is the reason.
Not yet. Postcode is what moves the score. Why postcode moves it, and what in the past made it matter, is the reason, and poking cannot reach that.
No. It tells you what moves the score, not what the model copied from the past that made postcode matter.
Right. Poking finds the lever. It does not find the reason the lever is wired that way, which is somewhere in the examples the model learned from, and in 175 billion numbers nobody can read.
No, because poking changed the model.
Not yet. The model is fixed; poking changes only the form. Every frame is the same model scoring a slightly different form.
Move the control.
A right to the logic2018
vsvs
175 billion numbers
This matters most where you have a right to argue: a refused loan, a stopped benefit, a bail decision, a job you did not get. In Europe a law that took effect in 2018 gives people a right to meaningful information about the logic of an automated decision about them. Whether a model can answer that is still open.
Each is something the bank could tell you. Is it a reason, a number, or a poke?
One card at a time. Tap the pile it belongs to.
Card 1 of 5
Your score was 61.
The rule says: income under £20,000 is refused.
When we changed your postcode, the score rose to 70.
Years at address counts for 0.3 in the model.
You missed two payments, and the rule is two missed payments means no.
Your score was 61. → A number
Not yet. A number. It is the result, not what led to it. Knowing the score is knowing you were refused, in different words.
The rule says: income under £20,000 is refused. → A reason
Not yet. A reason. A written rule carries its why in words, and you can argue with the £20,000.
When we changed your postcode, the score rose to 70. → A poke
Not yet. A poke. Somebody changed one thing and watched. It shows what moves the score, found from outside the model.
Years at address counts for 0.3 in the model. → A number
Not yet. A number. A weight says how much a clue counts, not why it counts. It is one of the millions, read out.
You missed two payments, and the rule is two missed payments means no. → A reason
Not yet. A reason. It names what you did and the rule it broke, which is something a person can check and dispute.
2A reason
++
2A number
++
1A poke
Right. Two were reasons, and both came from rules a person wrote. The numbers and the poke are all a learned model can offer: how much, and what moves it. Never because.
The modelhigh risk
↓→
Top factorsage, postcode, last visit
A hospital model flags a patient as high risk and the doctor asks it why. The screen prints: top factors, age, postcode, last visit. Is that the model's reason?
No. It is a list of what moved the score most, found by poking. The model holds no reason, only numbers.
Right. Top factors is the explore step printed on a screen: the levers that move the score. Why those levers are wired that way is in the examples, and nobody can read it out.
Yes. The model has printed its reasoning.
Not yet. A model has no reasoning to print. The list was made by changing inputs and watching the score, after the fact, by a separate program.
No, because a model cannot be wrong about a patient.
Not yet. Models are wrong about patients often, lesson 3. That is a different point. This one is that even when the model is right, it cannot say why.
Lesson complete
A model holds no reason, only numbers that add up. Poking finds what moves the score, never why.