In 2017 two researchers tested face software from three companies on 1,270 photos from six countries, and published the results in 2018. The software had one job: say man or woman. The company that did worst overall was wrong on light-skinned men less than 1 time in 100. On dark-skinned women, 35 times in 100.
Light-skinned men0.3 wrong in 100
Light-skinned women7 wrong in 100
Dark-skinned men12 wrong in 100
Dark-skinned women35 wrong in 100
The company that did worst overall, by group. Tested 2017.
The companies do not say which photos their systems learned from. But the public photo sets the whole field tests itself on are mostly light-skinned: one common set was 86 in 100 light, another 80 in 100. The faces seen least were the faces got wrong most. Nobody chose that. Nobody chose enough of the other photos either.
Tap the group the software was wrong about most.
Tap the part of the picture that answers it.
Light-skinned men0.3 wrong in 100
Light-skinned women7 wrong in 100
Dark-skinned men12 wrong in 100
Dark-skinned women35 wrong in 100
Dark-skinned women
Right. 35 wrong in 100, against under 1 for light-skinned men. The group the training photos held least of is the group the model knows least.
Dark-skinned men
Not yet. 12 in 100, the second-worst. The pattern runs the same way: fewer photos, more mistakes. The worst group is one bar further along.
Light-skinned women
Not yet. 7 in 100. Worse than the men, and far better than the dark-skinned women, because the photo sets held many light faces and fewer dark ones.
Light-skinned men
Not yet. Under 1 in 100. This is the group the training photos were full of, and it is the group the model is best on.
16 of 20
Light-skinned faces, each figure five of the 100
Made up and round.
Picture a training set of 100 faces, made up and round: 80 light-skinned and 20 dark-skinned. The model gets four light faces of practice for every dark one. It is not that it dislikes anyone. It has simply been shown one kind of face four times as often.
20 of 100
Dark-skinned faces in the set of 100
In that set of 100 photos, 80 are light-skinned and 20 are dark-skinned. How many light faces does the model see for each dark one?
light faces per dark face
Divide the bigger count by the smaller one.
Right. Four. Eighty split among twenty is four each. Four times the practice on one kind of face, and lesson 1 says practice is all a model has.
Move the control through a group's share of the training photos. The error figures are made up and round; the shape is the real one.
Wrong on that group30 in 100
Wrong on the biggest group3 in 100
The group's share of the photos: 2
The group is 2 in 100 of the training photos. The model is wrong on them about 30 times in 100. Less practice, more mistakes.
Wrong on that group20 in 100
Wrong on the biggest group3 in 100
The group's share of the photos: 5
The group is 5 in 100 of the training photos. The model is wrong on them about 20 times in 100. Less practice, more mistakes.
Wrong on that group12 in 100
Wrong on the biggest group3 in 100
The group's share of the photos: 10
The group is 10 in 100 of the training photos. The model is wrong on them about 12 times in 100. Less practice, more mistakes.
Wrong on that group6 in 100
Wrong on the biggest group3 in 100
The group's share of the photos: 25
The group is 25 in 100 of the training photos. The model is wrong on them about 6 times in 100. With a fair share of the photos, the errors come down to the level the other groups get.
Wrong on that group3 in 100
Wrong on the biggest group3 in 100
The group's share of the photos: 50
The group is 50 in 100 of the training photos. The model is wrong on them about 3 times in 100. With a fair share of the photos, the errors come down to the level the other groups get.
A group is 5 in 100 of the photos. Doubling it to 10 in 100 cuts the errors from 20 to 12. What is the plain lesson?
The model dislikes small groups.
Not yet. A model has no likes. It has practice, and it had less practice on the small group. Give it the photos and the gap closes.
The model is worst on whoever it saw least, and more photos of them is what brings the errors down.
Right. Each frame changes one thing, the share of photos, and the errors follow it down. Nothing about the maths changed between frames.
Errors get fixed in the maths, not in the photos.
Not yet. The maths was the same in every frame. The only thing that moved was the share of photos, and that alone took the errors from 30 to 3.
Move the control.
Police matching
++
Airport gates
++
Phone unlock
Why it mattered: this software was not grading a quiz. Police forces, airports and phone makers were buying face systems. One that is wrong 35 times in 100 on one group and once in 100 on another spreads its mistakes unevenly. The people it saw least are the ones it stops, locks out or mismatches most.
Put the chain in order, from the photos to the people.
Tap them in the order they happen, first to last. Each tap gets a number.
Right. Photos, practice, mistakes, people. Each link follows from the one before, and the last one lands on a person who was never in the photo set at all.
First test, 201721 wrong in 100
Second test, 20182 wrong in 100
One company's error on dark-skinned women.
After the study, all three companies put out new versions within seven months. Two of them said the fix was training on more varied photos; the third said little in public. In August 2018 one of the researchers tested again, with a colleague. One company's error on dark-skinned women had fallen from 21 in 100 to under 2.
Training photosalmost all light-skinned
↓→
A dark-skinned patient?
A hospital app reads photos of skin and flags possible cancers. It learned from photos taken at a clinic where almost every patient was light-skinned. A dark-skinned patient uses it. What is the risk?
None. Skin cancer looks the same on everyone to a computer.
Not yet. A computer does not know what skin cancer looks like. It knows what the photos it was shown looked like, and those were nearly all one kind of skin.
It will refuse to answer, because it knows what it has not seen.
Not yet. A model has no idea what it has not seen. It gives a score to every photo, confident or not, and the line turns the score into a flag.
It is more likely to be wrong on them, both ways: missing a real problem, or flagging a harmless mark.
Right. The app has had almost no practice on skin like theirs. Lesson 3: a wrong score can fall on either side of the line, so both kinds of mistake rise.
Lesson complete
A model is worst on whoever it saw least. The companies that explained their fix named more varied photos.