In 2014 Amazon began building a model to read job applications and score them one to five stars, the way it scored products. It learned from ten years of CVs the company had received. Most of those, at a technology firm, had come from men. What happened was reported in 2018.
Past hires, mostly men→The pattern: hired looks like this→A CV says women's→Marked down
Lesson 1: a model copies the pattern in its examples. The examples here were past hires. If the people hired were mostly men, then looks like a past hire and is a man point the same way. The model found that pattern because it was there. It marked down CVs with the word women's, as in women's chess club captain.
Past hiresmostly men
↓→
The modelprefers…
Nobody told the model to prefer men. Where did the preference come from?
From the engineers, who wrote it in.
Not yet. Lesson 1: nobody writes the rule in a model. The engineers supplied the examples. The preference was in the examples, and the model found it.
From the examples. The past hires it copied were mostly men.
Right. The model's only knowledge is the CVs it was shown and who was hired. A pattern in those hires became a pattern in the model, with nobody asking for it.
From the new applicants.
Not yet. The new applicants were being scored, not learned from. The model's habits were fixed before any of them applied.
Men800 past hires
Women200 past hires
Made up and round. Who was hired over ten years.
Here is the shape in made-up numbers. Picture 1,000 past hires: 800 men and 200 women. This year, equally good men and women apply. A model copying the past ranks the men higher, because men look more like the people who got hired. Nothing in the maths knows that the past was lopsided.
Equally good applicants of both kinds apply. Tap the group a model copying this past ranks higher.
Tap the part of the picture that answers it.
Men800 past hires
Women200 past hires
Men
Right. The model's idea of a good hire is the past hires, and 800 of 1,000 were men. An applicant who looks like them scores higher, however good the others are.
Women
Not yet. The model has no sense of fairness pulling the other way. It has 200 examples of hired women against 800 of hired men, and it ranks by what it saw.
Move the control through the share of men among past hires. The figures are made up and round; the model's tilt follows the past's.
Model ranks the man higher50 in 100
Model ranks the woman higher50 in 100
An equal man and an equal woman apply.
Men among past hires: 50
Half the past hires were men. The model has no tilt to copy: an equal man and an equal woman get the same rank.
Model ranks the man higher60 in 100
Model ranks the woman higher40 in 100
An equal man and an equal woman apply.
Men among past hires: 60
60 in 100 past hires were men. Looks like past hires now leans male, and the model ranks an equal man above an equal woman about 60 times in 100. The tilt in the model is the tilt in the past.
Model ranks the man higher70 in 100
Model ranks the woman higher30 in 100
An equal man and an equal woman apply.
Men among past hires: 70
70 in 100 past hires were men. Looks like past hires now leans male, and the model ranks an equal man above an equal woman about 70 times in 100. The tilt in the model is the tilt in the past.
Model ranks the man higher80 in 100
Model ranks the woman higher20 in 100
An equal man and an equal woman apply.
Men among past hires: 80
80 in 100 past hires were men. Looks like past hires now leans male, and the model ranks an equal man above an equal woman about 80 times in 100. The tilt in the model is the tilt in the past.
Model ranks the man higher90 in 100
Model ranks the woman higher10 in 100
An equal man and an equal woman apply.
Men among past hires: 90
90 in 100 past hires were men. Looks like past hires now leans male, and the model ranks an equal man above an equal woman about 90 times in 100. The tilt in the model is the tilt in the past.
At 50 in 100 men among past hires the model is even. What does that tell you about where the tilt comes from?
The tilt is the past's. A level past gives a level model, and a lopsided one is copied in.
Right. The only thing that changes between frames is the past. The model's tilt rises with it and vanishes when the past is even. The model added nothing of its own.
A lopsided past does not matter if the model is good at maths.
Not yet. The maths is what copies the past faithfully. Better maths copies the lopsidedness more exactly, not less.
The model is biased on its own, whatever the past.
Not yet. At 50 in 100 the model is even, so there is no tilt of its own. Everything it has was copied from the hires it was shown.
Move the control.
Ignore the word women's
↓→
Other clues remain
↓→
Team disbanded
Amazon's engineers tried to fix it by telling the model to ignore the word women's, and the names of two women-only colleges it had marked down. It also favoured words men used more often, such as executed. They could not be sure it had not found other clues. The team was disbanded.
Each is a set of examples a model could learn from. Does it record a decision people made, or a plain measurement?
One card at a time. Tap the pile it belongs to.
Card 1 of 6
Which loan applicants a bank said yes to, 2010 to 2020.
Temperatures recorded at the airport each hour.
Which pupils teachers put in the top set.
Where police made the most arrests.
How many millimetres of rain fell on each day.
Which CVs were given an interview.
Which loan applicants a bank said yes to, 2010 to 2020. → People's decisions
Not yet. Decisions. Every yes was a person at the bank choosing, and the model will copy their habits along with their judgement.
Temperatures recorded at the airport each hour. → Measurements
Not yet. A measurement. A thermometer does not decide anything; whatever habits the airport staff had are not in the number.
Which pupils teachers put in the top set. → People's decisions
Not yet. Decisions. Teachers chose, and whatever leaned on their choices leans on the model too.
Where police made the most arrests. → People's decisions
Not yet. Decisions. An arrest happens where police chose to look. The record is of their looking as much as of crime.
How many millimetres of rain fell on each day. → Measurements
Not yet. A measurement. Rain fell or it did not. Nobody's choice is folded into the count.
Which CVs were given an interview. → People's decisions
Not yet. Decisions. A person read each CV and chose. The model learns the choosing.
4People's decisions
vsvs
2Measurements
Right. Four of the six are records of people choosing. A model trained on them learns the choosing, including whatever the choosers got wrong. Only two are the world measured without anyone deciding.
People decided, years ago→A model copied them→It decides today, for strangers
This is why the computer decided is never the whole story. The computer copied. Whoever made the old decisions is still making them, through the pattern, years later, for people they never met. The past does not have to be cruel for this to go wrong. It only has to be lopsided.
Ten years of patrolsuneven
↓→
The model
↓→
Tomorrow's patrols?
A city trains a model on ten years of police records to predict where crimes will happen tomorrow. In those years, police patrolled some areas far more than others. Where will the model send them?
Evenly across the city. A computer has no favourites.
Not yet. It has no favourites of its own. It has the ones in its examples, and the examples favoured the areas that were patrolled most.
Mostly to the same areas. More patrols meant more recorded crime there, and the model copies the record.
Right. The record is of where police looked as much as of where crime was. The model sends them back to look where they looked, and the record grows there again.
Wherever crime really is. Records are records.
Not yet. The records are of arrests, and an arrest needs a police officer present. Where officers were thin, crime went unrecorded, and the model cannot see what nobody wrote down.
Lesson complete
A model trained on old decisions copies the habits of the people who made them, mistakes included.