Scored higher risk, among people who were not arrested again.
In 2016 reporters studied a score used by US courts to guess who would be arrested again. Among people who were not arrested again, 45 in 100 Black defendants had been scored higher risk. Among white defendants who were not, 24 in 100 had. The company said it was fair. The reporters said it was not. Both had numbers.
Black defendants45 in 100
White defendants24 in 100
False alarms: scored higher risk, not arrested again.
The company's point: any score, a 7 say, meant about the same chance of re-arrest for Black and white people. One kind of fair: a score means the same for everyone. The reporters' point: among people who stayed out of trouble, far more Black people were marked high risk. Another meaning: the same chance of a false alarm for everyone.
Same false alarmsfor each group
vsvs
Same meaning of a scorefor each group
Which meaning of fair were the reporters using?
A score means the same thing for each group.
Not yet. That was the company's meaning, and by their numbers it held. The reporters were measuring something else: who got wrongly flagged.
The same share of false alarms for each group.
Right. Their 45 and 24 are false alarms: people marked high risk who were not arrested again. The gap between those two numbers is what they called unfair.
The same number of people flagged in each group.
Not yet. Neither side was counting how many were flagged. One counted what a flag meant; the other counted who was flagged wrongly.
Group A non-payers40 in 100
Group B non-payers20 in 100
Made up and round.
Now the hard part, with made-up loan numbers. Two groups of 100 applicants. In Group A, 40 will not pay back. In Group B, 20. The same model, with the same line, catches 8 in 10 non-payers and wrongly flags 2 in 10 payers, in both groups. Equal treatment, exactly.
Payers60 in Group A
Non-payers40 in Group A
Group A has 40 non-payers and 60 payers. The model wrongly flags 2 in 10 payers. How many payers in Group A are wrongly flagged?
payers
Two in ten of the sixty payers.
Right. Twelve. In Group B there are 80 payers, so 16 are wrongly flagged. Same share, 2 in 10, in both: that is the reporters' kind of fair, held exactly.
Set each group's true rate of non-payers. Same model and line in both: 8 in 10 non-payers caught, 2 in 10 payers wrongly flagged. Computed from those rules.
A flag in Group A is right50 times in 100
A flag in Group B is right50 times in 100
Group A non-payers in 100: 20 · Group B non-payers in 100: 20
Group A: 20 non-payers in 100, Group B: 20. Same line in both; 2 in 10 payers wrongly flagged in each. A flag in A is right 50 times in 100; in B, 50. Equal rates, and a flag is worth the same in both.
A flag in Group A is right50 times in 100
A flag in Group B is right63 times in 100
Group A non-payers in 100: 20 · Group B non-payers in 100: 30
Group A: 20 non-payers in 100, Group B: 30. Same line in both; 2 in 10 payers wrongly flagged in each. A flag in A is right 50 times in 100; in B, 63. Same rule, same share of false alarms, and a flag means something different in each group.
A flag in Group A is right50 times in 100
A flag in Group B is right73 times in 100
Group A non-payers in 100: 20 · Group B non-payers in 100: 40
Group A: 20 non-payers in 100, Group B: 40. Same line in both; 2 in 10 payers wrongly flagged in each. A flag in A is right 50 times in 100; in B, 73. Same rule, same share of false alarms, and a flag means something different in each group.
A flag in Group A is right63 times in 100
A flag in Group B is right50 times in 100
Group A non-payers in 100: 30 · Group B non-payers in 100: 20
Group A: 30 non-payers in 100, Group B: 20. Same line in both; 2 in 10 payers wrongly flagged in each. A flag in A is right 63 times in 100; in B, 50. Same rule, same share of false alarms, and a flag means something different in each group.
A flag in Group A is right63 times in 100
A flag in Group B is right63 times in 100
Group A non-payers in 100: 30 · Group B non-payers in 100: 30
Group A: 30 non-payers in 100, Group B: 30. Same line in both; 2 in 10 payers wrongly flagged in each. A flag in A is right 63 times in 100; in B, 63. Equal rates, and a flag is worth the same in both.
A flag in Group A is right63 times in 100
A flag in Group B is right73 times in 100
Group A non-payers in 100: 30 · Group B non-payers in 100: 40
Group A: 30 non-payers in 100, Group B: 40. Same line in both; 2 in 10 payers wrongly flagged in each. A flag in A is right 63 times in 100; in B, 73. Same rule, same share of false alarms, and a flag means something different in each group.
A flag in Group A is right73 times in 100
A flag in Group B is right50 times in 100
Group A non-payers in 100: 40 · Group B non-payers in 100: 20
Group A: 40 non-payers in 100, Group B: 20. Same line in both; 2 in 10 payers wrongly flagged in each. A flag in A is right 73 times in 100; in B, 50. Same rule, same share of false alarms, and a flag means something different in each group.
A flag in Group A is right73 times in 100
A flag in Group B is right63 times in 100
Group A non-payers in 100: 40 · Group B non-payers in 100: 30
Group A: 40 non-payers in 100, Group B: 30. Same line in both; 2 in 10 payers wrongly flagged in each. A flag in A is right 73 times in 100; in B, 63. Same rule, same share of false alarms, and a flag means something different in each group.
A flag in Group A is right73 times in 100
A flag in Group B is right73 times in 100
Group A non-payers in 100: 40 · Group B non-payers in 100: 40
Group A: 40 non-payers in 100, Group B: 40. Same line in both; 2 in 10 payers wrongly flagged in each. A flag in A is right 73 times in 100; in B, 73. Equal rates, and a flag is worth the same in both.
Group A has 40 non-payers in 100, Group B has 20. Both get the same line, and the same share of payers is wrongly flagged. Is a flag worth the same in both?
Yes. The same rule was applied, so a flag means the same thing.
Not yet. The rule was the same and the groups were not. With fewer non-payers in B, the wrongly flagged payers make up a bigger share of B's flags, so a flag there is right less often.
No. In Group A a flag is right about 73 times in 100; in Group B about 50. The rule was the same for both groups. What a flag meant was not.
Right. In A, 32 non-payers caught against 12 payers wrongly flagged. In B, 16 against 16. The same line gives one kind of fair and takes away the other.
Yes, because the shares of false alarms are equal.
Not yet. Equal false alarms is one meaning of fair, and it holds here. The other meaning, a flag being right as often in both groups, does not. The frames show both at once.
Move both controls.
Same false alarms
vsvs
Same meaning of a flag
When the groups' true rates differ: pick one.
A flag in Group A is right 73 times in 100, in Group B only 50. To make a flag mean the same in both, one line must move. Then the two groups' payers get wrongly flagged at different rates. In 2016 researchers proved it: when two groups differ in the true rate, no score gives both kinds of fair.
Each is something a person calls fair. Which meaning is it?
One card at a time. Tap the pile it belongs to.
Card 1 of 5
A quarter of each group gets flagged.
A flag is right as often for one group as for the other.
A payer in either group has the same chance of being wrongly flagged.
The model catches the same share of non-payers in each group.
Half the loans go to each group.
A quarter of each group gets flagged. → Same numbers
Not yet. Same numbers. This counts how many are flagged in each group, and says nothing about whether the flags were right.
A flag is right as often for one group as for the other. → Same meaning
Not yet. Same meaning. This is about what a flag is worth once it is given: the company's meaning in the 2016 argument.
A payer in either group has the same chance of being wrongly flagged. → Same mistakes
Not yet. Same mistakes. This counts false alarms, the reporters' meaning: who gets wrongly caught, group by group.
The model catches the same share of non-payers in each group. → Same mistakes
Not yet. Same mistakes. Misses are the other kind of mistake, and asking for equal misses is the same family as asking for equal false alarms.
Half the loans go to each group. → Same numbers
Not yet. Same numbers. An equal count of yeses says nothing about whether each yes was deserved or each no was a false alarm.
2Same mistakes
++
1Same meaning
++
2Same numbers
Right. Three meanings, and they pull apart. The 2016 argument was between the first two. The third is a fourth thing again, and people reach for it when they have not said which of the others they mean.
Which fairness?→Who chose it?→Who carries the mistakes?
This is why is the algorithm fair has no yes or no. The real questions are which fairness, who chose it, and who carries the mistakes it leaves. The company and the reporters were each right about their own number. Neither number was the whole of fair.
The heada flag is equally accurate
vsvs
Pupils who were finemore flagged from one estate
A school's model flags pupils for a catch-up class. Among pupils who were in fact fine, far more from one estate were flagged than from elsewhere. The head says a flag is equally accurate for both. Can both be true?
No. If the flags are equally accurate, the false alarms must be equal too.
Not yet. The balance step showed the opposite: equal rules and different true rates give equal false alarms and unequal accuracy, or the other way round. Never both.
No. The head must be hiding something.
Not yet. Nothing needs hiding. Both statements can be true at once, which is the uncomfortable thing the lesson is about.
Yes. Equal accuracy of a flag and equal false alarms cannot both hold when the groups' true rates differ.
Right. This is the 2016 argument in a school. If more pupils from the estate really do struggle, a flag that means the same for both groups must wrongly catch more of the estate's fine pupils.
Lesson complete
Fair has more than one meaning. When two groups differ, one score cannot give both at once.