In March 2018 a self-driving test car hit and killed a woman crossing a road at night in Tempe, Arizona. The car's software saw something 5.6 seconds before the crash and kept changing its mind about what it was. The car's own factory emergency brake had been switched off. The safety driver behind the wheel was looking at her phone.
Engineers→The company→The driver→The state
Many people had a part in that death. The engineers who wrote software that kept changing its mind. The company that switched the brake off, and put one person alone in the car for hours. The driver who looked down. The state that allowed the test. Each one did a part. None of them did all of it.
Engineers→The company→The driver→The state
How many separate people or groups in that chain did something that helped cause the crash?
Several. Each did a part, and none of them did all of it.
Right. Take any one link out and the crash might not have happened. That is true of every link, which is what makes the chain hard to answer for.
One. The driver, who was not watching.
Not yet. The driver did a part. A watching driver might have braked. So might a factory brake that had not been switched off, or software that had made up its mind.
None. The computer did it.
Not yet. The computer did what people built it to do and left it able to do. A machine is in the chain. It is not a hand.
Of everyone in that chain, how many were charged with a crime?
Pick one. Nothing is scored.
The companynot charged
vsvs
The drivercharged
One.One: the driver. In 2019 the prosecutor decided the company would not be charged. In 2023 the driver pleaded guilty to a lesser offence and got three years' probation. Guesses tend to land on none or on the company. The law found the hand it could see.
The court
↓→
One driverprobation
Here is what happened in court. In 2019 the prosecutor decided the company would not be charged. In 2020 the driver was charged, and in 2023 she pleaded guilty to a lesser offence and was given three years' probation. Of the whole chain, the law held one person: the one at the end of it.
Move the control through the number of hands in the chain. The share each did is split evenly, which is made up. Computed.
Each hand's share100 in 100
The harm100 in 100
Hands in the chain: 1
One hand. Whatever happens, this person did it, and they know it.
Each hand's share50 in 100
The harm100 in 100
Hands in the chain: 2
2 hands. Each did about 50 in 100 of the work, and each can honestly say so. The harm is still the whole 100.
Each hand's share33 in 100
The harm100 in 100
Hands in the chain: 3
3 hands. Each did about 33 in 100 of the work, and each can honestly say so. The harm is still the whole 100.
Each hand's share20 in 100
The harm100 in 100
Hands in the chain: 5
5 hands. Each did about 20 in 100 of the work, and each can honestly say so. The harm is still the whole 100.
Each hand's share12 in 100
The harm100 in 100
Hands in the chain: 8
8 hands. Each did about 12 in 100 of the work, and each can honestly say so. The harm is still the whole 100.
With eight hands, each one did an eighth. What happens to how responsible each person feels?
It shrinks for each person. The harm does not shrink.
Right. Each of eight can truthfully say I did an eighth. The woman who was killed was killed once, wholly. The shares add up to the harm; the feelings do not.
It stays the same for each person.
Not yet. The bar for each hand falls in every frame, from 100 to 12. Only the harm stays at 100.
It grows. More people means more care.
Not yet. More people means more chances for somebody else to be the one who checks. Each hand's share, and each hand's sense of it, gets smaller as the chain gets longer.
Move the control.
Many small acts→Each looks too small→The machine gets pointed at→Nobody answers
People call this the problem of many hands. The phrase just means that when a harm is made of many small acts, each act looks too small to be the cause. A machine makes it worse. It adds a link that cannot be blamed at all, and every hand can point at it.
Put the chain in the order it happened.
Tap them in the order they happen, first to last. Each tap gets a number.
Software→Brake off→Alone in the car→Phone→Crash
Right. Software, brake, shift, phone, crash. The last link before the crash is the driver, which is why the law found her. The links before her were decided months earlier, in offices.
The investigators' reportthe whole chain
vsvs
The courtone driver
Compare a plane crash. Investigators do not ask who to blame first. They ask what each part of the system did, and publish it. The Arizona crash got that kind of report too. It named the driver's phone, the software, the switched-off brake, the lone shift, and the company's way of running safety. The court named the driver.
Who built the model→Who bought it→Who set 200 a day→The radiologist→The missed scan
A hospital model misses a cancer on a scan, and the radiologist who sees 200 scans a day signed it off. The hospital says the radiologist is responsible. From this lesson, what is missing from that answer?
The rest of the chain: who built the model, who bought it, who set 200 scans a day. The last hand is the easiest to see, not the only one.
Right. The radiologist did a part. So did the people whose decisions put a tired person and an imperfect model in the same room. Naming the last hand is where a chain usually stops being looked at.
Nothing. The person who signed is responsible.
Not yet. They are responsible for their part. The hospital's answer uses that part to close the question before the other hands are counted.
The model is responsible, and nobody else.
Not yet. The model is the link that cannot answer. Pointing at it is how every hand in the chain gets to point away from itself.
Lesson complete
Many hands make a harm, and each looks too small to be the cause. The last hand gets blamed.