Picture a clerk at a tax office who makes one wrong judgement a day. In a year about 250 families get a wrong letter. Now picture the same judgement written into a computer that checks every claim in the country. The same mistake lands on every family that fits it, in the same week.
Not Dutch, or a second nationalitycounted against you
↓→
Repay it all, now
↓→
Homes lost
In the Netherlands, from about 2013, the tax office used a system to flag childcare benefit claims as likely fraud. Not being Dutch, or having a second nationality, counted against you. Flagged parents had to repay years of benefit at once, often tens of thousands of euros. Families lost homes and jobs. In January 2021 the government resigned over it.
20 of 20
Each envelope stands for 100,000 claims
Made up and round.
One rule is applied to 2 million claims. It is wrong 1 time in 100. How many wrong letters go out?
wrong letters
One in a hundred of two million.
Right. Twenty thousand. A clerk would take eighty years to make that many wrong calls. The rule made them in the time it takes to run.
Slide to the number of parents the Dutch government counted as wrongly treated by the system, in thousands.
Slide the marker to where you think the answer is, then press Lock. Nothing is scored.
12
your guess · 0 thousand parents
the answer · 26 thousand parents
02550
thousand parents
1Ten10 thousand parents
2Forty40 thousand parents
Your guess: —
Twenty-six thousand by the government's own count, and later counts ran higher. Each of those families was told, alone, that they owed the money. The pattern across all of them took years to be seen.
A rule is written→Thousands of letters→Each family alone→Years later, counted together
Why nobody notices in time. A clerk's mistake is seen by the clerk, the family, and the next clerk who reads the file. A rule's mistake is seen by nobody at the office; it is a line in a program. Each family thinks it is alone. The pattern only shows when thousands of alone families are counted together.
Move the control through how many claims the rule checks a year. It is wrong 1 time in 100 in every frame. Computed.
Wrong letters from the rule100 a year
Wrong letters from one clerk250 a year
Claims checked a year: 10,000
10,000 claims checked a year, wrong 1 time in 100: 100 wrong letters. One clerk making one wrong call a day sends about 250. Still fewer than the clerk.
Wrong letters from the rule1,000 a year
Wrong letters from one clerk250 a year
Claims checked a year: 100,000
100,000 claims checked a year, wrong 1 time in 100: 1,000 wrong letters. One clerk making one wrong call a day sends about 250. Four times the clerk already.
Wrong letters from the rule10,000 a year
Wrong letters from one clerk250 a year
Claims checked a year: 1 million
1 million claims checked a year, wrong 1 time in 100: 10,000 wrong letters. One clerk making one wrong call a day sends about 250. The rule's error rate has not changed between frames. Only the number of people it reaches.
Wrong letters from the rule100,000 a year
Wrong letters from one clerk250 a year
Claims checked a year: 10 million
10 million claims checked a year, wrong 1 time in 100: 100,000 wrong letters. One clerk making one wrong call a day sends about 250. The rule's error rate has not changed between frames. Only the number of people it reaches.
The rule's error rate never changes. What does change as it checks more claims?
The error rate. A busier system makes more mistakes per claim.
Not yet. The rate was fixed at 1 in 100 in every frame. A program does not get tired. What grew was the count of people, not the carelessness.
The number of people hit. It goes up ten times for every ten times more claims.
Right. One in 100 of ten thousand is 100; of ten million it is 100,000. The rule is no more careless at the end than at the start. It simply reaches more people.
Nothing. One in 100 is one in 100.
Not yet. One in 100 of a bigger number is a bigger number. The share stayed; the people did not.
Move the control.
A year's payspread evenly over fortnights
↓→
A debt letter470,000 of them
Australia did it with a different rule. From 2016 its welfare office let a computer compare a person's yearly tax income with their fortnightly benefit claims, by spreading the year evenly across the fortnights. People whose work came and went were told they owed money they did not. About 470,000 debts were raised that way before it was ruled unlawful.
Put the Dutch story in the order it happened.
Tap them in the order they happen, first to last. Each tap gets a number.
Rule→Letters→Alone→Counted→Resigned
Right. Rule, letters, alone, counted, resigned. The third step is the one that takes years, and it is the one a single clerk's mistake never has, because somebody at the office would have seen the clerk.
Decided at midnight
↓→
Letter printed
↓→
Undone one case at a time
Speed is the other half. A clerk makes a mistake and the family can argue with the clerk. A system makes it at midnight and the letter is printed before anyone is awake. To undo it, every case has to be found and reopened by hand. Robodebt took more than three years to stop, and longer to repay.
A personone mistake, seen
vsvs
A ruleevery matching case, unseen
A council buys a model to spot benefit fraud and lets it decide small cases on its own. A councillor says: a person would make mistakes too, so what is the difference? What is the best answer?
None. A mistake is a mistake.
Not yet. One wrong letter and twenty thousand identical wrong letters are not the same event, even if each letter is just as wrong. The rule multiplies.
A person's mistake is one case, seen by someone. The rule's mistake is every case that fits, at once, seen by nobody until they are counted.
Right. The councillor is right that a person errs. The difference is in the spread and the seeing: a rule's error reaches everyone it matches in one run, and nobody at the office watches it happen.
The model makes fewer mistakes, so it is always safer.
Not yet. It may well make fewer per case. The lesson is about what happens to the ones it does make: they land everywhere that fits, in the same week, with nobody in the room.
Lesson complete
A person's mistake is one case. A rule's mistake is every case that fits, at once, unseen.