Picture a benefit office where a model flags a claim and a person must click agree before a letter goes out. That is what people mean by a human in the loop. The phrase just means a person sits between the machine and the decision. This lesson asks what the person actually does there.
Checking all 400 properly67 hours
Clicking agree on all 4001 hour
Made up and round. 400 claims, ten minutes against ten seconds each.
Here is the problem, in made-up numbers. The person has 400 claims to clear today. The model is right about 95 times in 100. Checking a claim properly takes ten minutes. Clicking agree takes ten seconds. After a week of the model being right, the finger learns where the agree button is.
400 claims
vsvs
Seven hours
The person has 400 claims and a seven-hour day. Checking one properly takes ten minutes. What happens?
They check them all, by working faster.
Not yet. Ten minutes is what a check takes. A check done in one minute is not the same check; it is a glance, and a glance is what the agree button gets.
The model checks them.
Not yet. The model is the thing being checked. It cannot be its own second pair of eyes.
They cannot check them all. Most get clicked through.
Right. Four hundred checks need 67 hours. The day has seven. The arithmetic decides what the person does before the person does.
Machine is usually right→Person stops checking→The wrong 5 in 100 go through
People who study this call it automation bias. Bias here just means a lean: going along with the machine because it is usually right. Pilots do it with autopilots and doctors with scan software. The 5 in 100 the model gets wrong are the cases the person was there to catch, and they look like all the others.
Move the control through how many claims each person must clear in a seven-hour day. A real check takes ten minutes. Computed.
Minutes each claim gets21 minutes
Minutes a real check needs10 minutes
Claims per person per day: 20
20 claims in a seven-hour day: 21.0 minutes each. A real check needs ten. There is time to check.
Minutes each claim gets8.4 minutes
Minutes a real check needs10 minutes
Claims per person per day: 50
50 claims in a seven-hour day: 8.4 minutes each. A real check needs ten. There is not. The person is now a finger on the agree button.
Minutes each claim gets4.2 minutes
Minutes a real check needs10 minutes
Claims per person per day: 100
100 claims in a seven-hour day: 4.2 minutes each. A real check needs ten. There is not. The person is now a finger on the agree button.
Minutes each claim gets2.1 minutes
Minutes a real check needs10 minutes
Claims per person per day: 200
200 claims in a seven-hour day: 2.1 minutes each. A real check needs ten. There is not. The person is now a finger on the agree button.
Minutes each claim gets1.1 minutes
Minutes a real check needs10 minutes
Claims per person per day: 400
400 claims in a seven-hour day: 1.1 minutes each. A real check needs ten. There is not. The person is now a finger on the agree button.
At what load does the person stop being a check?
Somewhere past about 40 a day, when each claim gets less than the ten minutes a check needs.
Right. At 20 a day each claim gets 21 minutes, more than enough. At 50 it gets 8.4, already short. The person did not change; the day did.
At any load. A person was never a check.
Not yet. At 20 a day the person has time to check every claim twice over. The check is real there. It is the load that takes it away.
Never. A person is always a check.
Not yet. A person with about a minute per claim is not checking anything. The loop still has a human in it, and the human has no time to be one.
Move the control.
An officer checked with the employer
↓→
Step removed, 2016
↓→
Letter goes straight out
Australia's Robodebt did this in the open. Before the scheme, an officer looked into a mismatch before raising a debt, often by asking the employer for pay records. From 2016 that step went: the letter went straight out, and the person had to prove the debt wrong. The royal commission later called that shift unfair.
Set each person's daily load and how often the model is wrong. A person checking 50 a day catches 9 in 10 wrong letters; at 100, half; at 200, 1 in 10. Made up and round.
Wrong letters caught9 in 1,000 claims
Wrong letters sent1 in 1,000 claims
Claims per person per day: 50 · Model wrong, in 100: 1
50 claims a day per person, model wrong 1 in 100. Of 10 wrong letters in a thousand claims, the person catches 9 and 1 go out.
Wrong letters caught45 in 1,000 claims
Wrong letters sent5 in 1,000 claims
Claims per person per day: 50 · Model wrong, in 100: 5
50 claims a day per person, model wrong 5 in 100. Of 50 wrong letters in a thousand claims, the person catches 45 and 5 go out.
Wrong letters caught90 in 1,000 claims
Wrong letters sent10 in 1,000 claims
Claims per person per day: 50 · Model wrong, in 100: 10
50 claims a day per person, model wrong 10 in 100. Of 100 wrong letters in a thousand claims, the person catches 90 and 10 go out.
Wrong letters caught5 in 1,000 claims
Wrong letters sent5 in 1,000 claims
Claims per person per day: 100 · Model wrong, in 100: 1
100 claims a day per person, model wrong 1 in 100. Of 10 wrong letters in a thousand claims, the person catches 5 and 5 go out.
Wrong letters caught25 in 1,000 claims
Wrong letters sent25 in 1,000 claims
Claims per person per day: 100 · Model wrong, in 100: 5
100 claims a day per person, model wrong 5 in 100. Of 50 wrong letters in a thousand claims, the person catches 25 and 25 go out.
Wrong letters caught50 in 1,000 claims
Wrong letters sent50 in 1,000 claims
Claims per person per day: 100 · Model wrong, in 100: 10
100 claims a day per person, model wrong 10 in 100. Of 100 wrong letters in a thousand claims, the person catches 50 and 50 go out.
Wrong letters caught1 in 1,000 claims
Wrong letters sent9 in 1,000 claims
Claims per person per day: 200 · Model wrong, in 100: 1
200 claims a day per person, model wrong 1 in 100. Of 10 wrong letters in a thousand claims, the person catches 1 and 9 go out.
Wrong letters caught5 in 1,000 claims
Wrong letters sent45 in 1,000 claims
Claims per person per day: 200 · Model wrong, in 100: 5
200 claims a day per person, model wrong 5 in 100. Of 50 wrong letters in a thousand claims, the person catches 5 and 45 go out.
Wrong letters caught10 in 1,000 claims
Wrong letters sent90 in 1,000 claims
Claims per person per day: 200 · Model wrong, in 100: 10
200 claims a day per person, model wrong 10 in 100. Of 100 wrong letters in a thousand claims, the person catches 10 and 90 go out.
The model is wrong 5 in 100 times. The office doubles the load from 100 to 200 a day. What happens to the wrong letters sent, per thousand claims?
They go from 25 to 45. The model did not change. The check did.
Right. Fifty wrong letters either way. At 100 a day the person catches half of them; at 200, one in ten. The extra twenty wrong letters are the cost of the doubled load.
They stay at 50. The model's errors are the model's.
Not yet. Fifty is how many the model gets wrong. How many go out depends on the person, and the person catches fewer when they have half the time.
They fall, because more claims get seen.
Not yet. More claims get clicked, not seen. Each gets half the time it had, and the catch rate falls from half to one in ten.
Move both controls.
Time to check→Free to say no→No sometimes happens
So a human in the loop is real when three things hold. The person has time to check. The person can say no without trouble. And no sometimes happens, and is counted. Take away any one of the three, and the person is only signing what the machine decided.
Each is an office with a person in the loop. Is the person a real check, or a signature?
One card at a time. Tap the pile it belongs to.
Card 1 of 5
Forty claims a day, and every override is reviewed and the officer thanked.
Four hundred claims a day, ten seconds each.
The person may say no, but every no must be explained to a manager in writing.
The model's suggestion is hidden until the person has written down their own view.
The person has not said no once in a year.
Forty claims a day, and every override is reviewed and the officer thanked. → A real check
Not yet. A real check. There is time, and saying no is welcomed rather than punished.
Four hundred claims a day, ten seconds each. → A signature
Not yet. A signature. Ten seconds is a glance. The arithmetic has removed the check before the person sits down.
The person may say no, but every no must be explained to a manager in writing. → A signature
Not yet. A signature. A no costs an hour and a yes costs nothing, so the lean towards yes is built into the desk.
The model's suggestion is hidden until the person has written down their own view. → A real check
Not yet. A real check. The person decides first, so there is nothing to lean on until they have done their own thinking.
The person has not said no once in a year. → A signature
Not yet. A signature. A check that has never fired is not firing. Either the model is perfect or the person has stopped looking, and no model is perfect.
2A real check
vsvs
3A signature
Right. Two of the five are checks: time, and the freedom to disagree, and a no that is sometimes made. The other three keep the person and lose the checking.
300 reports a day
↓→
A radiologist signs
A hospital says its scan software never decides alone: a radiologist signs every report. The radiologists each sign 300 reports a day. From this lesson, what is the hospital's claim worth?
Everything. A doctor signed, so a doctor decided.
Not yet. A signature at 300 a day is automation bias with a pen. The doctor's name is on the decision; the time to make it was not.
Nothing, because software should never be used on scans.
Not yet. The lesson says nothing against the software. It says the person beside it needs time, freedom to disagree, and a record of disagreeing.
Little. At 300 a day the signature is a lean on the machine, not a check of it.
Right. A seven-hour day gives each report under a minute and a half. The radiologist is in the loop by name. Whether they can catch the software's misses at that speed is the question the claim skips.
Lesson complete
A human in the loop is real when the person has time, is free to say no, and sometimes does.