In 2024 the European Union passed the first big law about these machines. Not every use is the same risk. A spam filter and a model that scores asylum claims are both models, lesson 1. The law sorts uses into four tiers by how much harm a mistake can do, and the duties get heavier up the tiers.
Social scoring
++
Emotion reading in class
++
Scraping faces
The top tier is banned outright. Scoring citizens on their behaviour, the way a government might rank its people. Reading workers' or pupils' emotions from their faces, unless for a medical or safety reason. Scraping faces off the internet to build a search for anyone's face. These were judged too harmful however carefully they were built.
A spam filterminimal
vsvs
An asylum modelhigh risk
A spam filter and an asylum-scoring model are both models. Why does the law treat them differently?
Because spam filters are older.
Not yet. Age has nothing to do with the tier. A brand-new spam filter is minimal risk, and a decades-old scoring system for asylum would be high.
By how much harm a mistake can do to a person. The tier follows the use, not the maths.
Right. Lesson 3: every model makes mistakes. The law asks what a mistake costs the person on the wrong end of it, and a wrongly refused asylum claim costs more than a lost email.
Because the asylum model is more complicated.
Not yet. The two might be the same size and shape. The law does not look inside the model; it looks at what the model is pointed at.
High riskduties
↓→
Limitedmust say so
↓→
Minimalno rules
The second tier is high risk, where most of this course has lived: hiring, exam grading, loans, benefits, policing, border checks. Allowed, with duties: tested before use, watched after, a person able to step in, records kept, people told. Below that, limited risk: a chatbot must say it is one, a fake video must be labelled. Below that, nothing.
Each is a use of a model. Which tier does the law put it in?
One card at a time. Tap the pile it belongs to.
Card 1 of 6
A government ranks its citizens by how they behave.
A model grades school exams.
A shop's chatbot answers questions and says it is a bot.
A system watches pupils' faces in class to spot boredom.
A bank's model scores loan applications.
A video game's enemies learn how you play.
A government ranks its citizens by how they behave. → Banned
Not yet. Banned. Social scoring is named in the top tier. No amount of care makes it allowed.
A model grades school exams. → High risk
Not yet. High risk. Education and exam assessment are listed as high risk: allowed, with testing, records and a person able to step in.
A shop's chatbot answers questions and says it is a bot. → Limited or none
Not yet. Limited. The duty is to say so, and the chatbot has. Nothing further is asked of it.
A system watches pupils' faces in class to spot boredom. → Banned
Not yet. Banned. Reading emotions from faces in schools and workplaces is in the top tier, next to social scoring.
A bank's model scores loan applications. → High risk
Not yet. High risk. Credit scoring is listed: a wrong score shuts a person out of a flat or a car, so the duties apply.
A video game's enemies learn how you play. → Limited or none
Not yet. None. Nobody is harmed by a cleverer goblin, so the law has nothing to say about it.
2Banned
++
2High risk
++
2Limited or none
Right. Two uses are banned, two carry duties, two are left alone. The sort was by the harm a mistake does to a person, which is the whole of the law's idea.
35 million euros
vsvs
7 in 100 of worldwide sales
Whichever is larger.
Breaking the ban carries the biggest fine: 35 million euros, or 7 in every 100 euros the company took in worldwide that year, whichever is larger. For a company taking in 100 billion, that is 7 billion. That clause is aimed at the giants. A fine that is small change for a giant does not change what the giant does.
Move the control through a company's yearly sales worldwide. The fine for a banned use is the larger of 35 million euros and 7 in 100 of sales. Computed.
Fixed: 35 million35 million euros
7 in 100 of sales7 million euros
Yearly sales worldwide, euros: 100 million
Sales of 100 million euros a year. Seven in 100 is 7 million; the fixed fine is 35 million. The larger counts, so the fixed 35 million applies.
Fixed: 35 million35 million euros
7 in 100 of sales70 million euros
Yearly sales worldwide, euros: 1 billion
Sales of 1 billion euros a year. Seven in 100 is 70 million; the fixed fine is 35 million. The larger counts, so the share applies: 70 million.
Fixed: 35 million35 million euros
7 in 100 of sales700 million euros
Yearly sales worldwide, euros: 10 billion
Sales of 10 billion euros a year. Seven in 100 is 700 million; the fixed fine is 35 million. The larger counts, so the share applies: 700 million.
Fixed: 35 million35 million euros
7 in 100 of sales7,000 million euros
Yearly sales worldwide, euros: 100 billion
Sales of 100 billion euros a year. Seven in 100 is 7,000 million; the fixed fine is 35 million. The larger counts, so the share applies: 7,000 million.
Fixed: 35 million35 million euros
7 in 100 of sales35,000 million euros
Yearly sales worldwide, euros: 500 billion
Sales of 500 billion euros a year. Seven in 100 is 35,000 million; the fixed fine is 35 million. The larger counts, so the share applies: 35,000 million.
At 100 million in sales the fixed 35 million is larger. At 1 billion, which fine applies, and why is the law written that way?
The share: 70 million, because it is larger. The share keeps the fine big for big companies.
Right. Seven in 100 of a billion is 70 million, more than 35. A fixed fine shrinks as the company grows; a share grows with it, so the giant feels it too.
The fixed 35 million. It is always the fixed amount.
Not yet. Fixed is the floor, not the rule. At a billion in sales the share is 70 million, and the law takes the larger.
Both, added together.
Not yet. The law says whichever is larger, not both. At a billion that is the 70 million share on its own.
Move the control.
One model fixed→Every model in that use→Before it is switched on
A law is a different kind of fix. Better photos fix one model. A different label fixes one model. A law says: for this use, these duties, for everyone, before you start. It is slow and argued over. It is also the only fix so far that does not depend on the company choosing it.
Fixed fine35 million euros
7 in 100 of 50 billion3,500 million euros
A company with 50 billion euros of worldwide sales uses a banned system. The fine is the larger of 35 million or 7 in 100 of sales. How much, in millions of euros?
million euros
Seven in 100 of 50,000 million.
Right. 3,500 million, or three and a half billion. One hundred times the fixed fine. For a company that size, the fixed 35 million would be a rounding error.
A British company
↓→
Spanish schoolsfaces read in class
A British company sells Spanish schools an app that reads pupils' faces to spot boredom or anger. It says the law does not apply because the company is not in the EU. From this lesson, which is right?
Neither. The use is high risk, so duties apply but it is allowed.
Not yet. Reading emotions from faces in schools is named in the banned tier. The duties of the high-risk tier do not come into it.
The company. Laws stop at the border of the country that made them.
Not yet. This law is written to reach anyone selling into the EU, the way a food law reaches a foreign tin on a Spanish shelf. The company's address does not move the school.
The law follows the use inside the EU. A banned use in a Spanish school is banned whoever built it.
Right. The law applies to anyone putting a system to use in the EU, wherever the company sits. And reading pupils' emotions in school is in the banned tier, not merely the high one.
Lesson complete
The first big law sorts uses by the harm a mistake can do, and fines by the company's size.