In 2019 researchers studied a model used by US hospitals to pick which patients should get extra care. It was meant to find the sickest. But how sick is hard to count. So the model had been trained to predict how much each patient's care would cost. Cost is a number on a bill. Sickness is not.
The thing you mean→The thing you can count→The label on every example
Lesson 2 was about the number a machine is told to raise. This is the same move on the other side: the answer written on each example. People call that the label. The label is meant to be the truth about the example. Often it is the nearest thing anybody could count.
The labelcost
↓→
The hope?
The model's label was how much this patient's care cost. What were the hospitals hoping it stood for?
How sick the patient was.
Right. Cost was chosen because sicker patients usually cost more. Usually is the word the rest of the lesson is about.
How rich the patient was.
Not yet. Rich was not the aim, though money does creep in: how much is spent on a patient depends on more than how sick they are.
How long the patient would live.
Not yet. The programme was about extra care now, for people who were sick now. Cost was standing in for sickness, not for years ahead.
The same score
↓→
White patientsfewer illnesses
↓→
Black patientsabout a quarter more
From the 2019 study.
Here is where it broke. In the US, Black patients at the same level of illness had cost less: care was harder for them to reach, and less of it was given. The model saw lower cost and read it as less sick. At any given score, the Black patients were sicker, with about a quarter more long-term illnesses.
Under the cost label, slide to the share of patients flagged for extra care who were Black, in 100.
Slide the marker to where you think the answer is, then press Lock. Nothing is scored.
123
your guess · 0 in 100
the answer · 18 in 100
0 in 10050 in 100100 in 100
1A quarter25 in 100
2Half50 in 100
3Three quarters75 in 100
Your guess: —
Fewer than one in five. The study worked out what the share would be if the model had no gap between Black and white patients at the same sickness: about 47 in 100. Counted a different way, at the very top of the risk list, the share under the cost label was 14 in 100. The next steps start from that number.
Two groups, equally sick, in a made-up town. Move the control through how much less is spent on the second group at the same illness.
First group flagged50 of 100 sick people
Second group flagged50 of 100 sick people
Less spent on the second group: 0
The same is spent on both groups. The cost label and the sickness line up, and the model flags 50 of each 100 sick people in each group.
First group flagged50 of 100 sick people
Second group flagged40 of 100 sick people
Less spent on the second group: 10
10 in 100 less is spent on the second group at the same illness. The cost label reads them as less sick: 40 flagged, not 50. The group is exactly as sick as before.
First group flagged50 of 100 sick people
Second group flagged32 of 100 sick people
Less spent on the second group: 20
20 in 100 less is spent on the second group at the same illness. The cost label reads them as less sick: 32 flagged, not 50. The group is exactly as sick as before.
First group flagged50 of 100 sick people
Second group flagged25 of 100 sick people
Less spent on the second group: 30
30 in 100 less is spent on the second group at the same illness. The cost label reads them as less sick: 25 flagged, not 50. The group is exactly as sick as before.
The two groups are equally sick. Spending 30 in 100 less on one makes the model flag half as many of them. Who is the model actually finding?
Nobody. The model is broken and flags at random.
Not yet. The flags are not random. They fall exactly as the spending falls, which means the model is working perfectly on the label it was given.
The patients who cost most, which is not the same as the sickest.
Right. The label was cost, so cost is what the model learned to find. Where cost and sickness come apart, the model follows cost, because that is the only truth it was given.
The sickest patients, as designed.
Not yet. Both groups are equally sick in every frame, and the flags still fall for one of them. A model finding the sickest would flag the same number in each.
Move the control.
14 in 100Cost as the label
vsvs
27 in 100Illness as the label
The fix was not better maths. They changed the label. Trained to predict illnesses instead of money spent, the same shape of model nearly doubled the share of Black patients in its highest-risk group: 14 in 100 became 27. A later label mixing illness and cost, built with the model's seller, removed 84 in 100 of the gap.
Each is a label a model is trained on. Is it the thing people meant, or a stand-in that can drift away from it?
One card at a time. Tap the pile it belongs to.
Card 1 of 6
Did this patient die within a year.
Was this person arrested again within two years.
Did this parcel arrive.
Was this employee promoted within three years.
How many clicks this headline got.
Did this bridge fall down.
Did this patient die within a year. → The thing itself
Not yet. The thing. For a model meant to predict deaths, the label is the event itself, with nothing between.
Was this person arrested again within two years. → A stand-in
Not yet. A stand-in. The hope is committed another crime. An arrest needs a police officer present, so the label also counts where police looked.
Did this parcel arrive. → The thing itself
Not yet. The thing. Arrived is what everybody meant, and the label says exactly that.
Was this employee promoted within three years. → A stand-in
Not yet. A stand-in. The hope is did good work. Promotions also follow who the managers liked and who asked.
How many clicks this headline got. → A stand-in
Not yet. A stand-in. The hope is a good story. Clicks also rise for a headline that is misleading or that makes people angry.
Did this bridge fall down. → The thing itself
Not yet. The thing. There is no gap between a bridge falling and the label that says it fell.
3The thing itself
vsvs
3A stand-in
Right. Three labels are the event people meant. Three are a nearby count that usually moves with it. A model trained on the second kind becomes very good at the nearby count.
The model is wrong for one group→Look at the label→For whom does the count drift?
So when a machine keeps getting one group wrong, the first place to look is the label. The question is not whether the maths is right. It is what they counted, and which group that count misses. At those hospitals the maths was right. The thing being counted was money spent.
The court's model
++
Label: arrested again
++
Uneven patrols
A court uses a model to score how likely a person is to commit another crime. Its label was arrested again within two years. In that city some neighbourhoods are patrolled far more than others. What does the model actually learn?
Who is likely to be arrested again, which depends on where police look, not only on who commits crimes.
Right. The label is arrest, so arrest is what the model learns to predict. In a heavily patrolled area the same act is more likely to become an arrest, and the model learns that too.
Nothing about neighbourhoods, since it was not given addresses.
Not yet. Lesson 4: a pattern spread across many clues finds its way to the thing you hid. Where a person lives shows up in other clues, and the model uses them.
Who commits crimes, because an arrest means a crime.
Not yet. An arrest means an officer was there and acted. Many crimes bring no arrest, and some arrests bring no conviction. The label is the arrest, not the act.
Lesson complete
A model learns the label it is given. If the label is a stand-in, it finds the stand-in.