Day Lila

14-minute course

Course · Start

Tech · 14-minute course

Why do the biggest outages start with a fix?

In one week Apple's iCloud and the NSA's secret network went down after their own updates and maintenance, and an overloaded new router halted US flights. Google's engineers found roughly 70% of outages start with a change.

Made from the news of Wednesday, 18 November 2026.

Swipe up to start ↑Scroll down to start ↓

A week of outages that came from inside

Swipe up to start ↑Scroll down to start ↓

In the news · 18 Nov 2026

On Tuesday, Apple's iCloud sign-in, App Store and Apple Music went down. Sign-in was hit for all users.

See the story →

In the news · 18 Nov 2026

“an internal update”
Apple

Apple said "an internal update" caused the outages, not an attack.

See the story →

In the news · 18 Nov 2026

It was the 4th big system failure in 4 working days, after Microsoft, the FAA, the US air traffic agency, and JPMorgan Chase.

See the story →

In the news · 18 Nov 2026

And the flights?

On Friday an overloaded router in southern Arkansas stopped US departures for more than 4 hours, the longest halt since 9/11.

See the story →

In the news · 18 Nov 2026

The cause is not yet known.

That router was part of a project moving the FAA's network from copper wires to fibre-optic cable. The FAA has not yet found why it overloaded.

See the story →

In the news · 18 Nov 2026

“temporary degradation during scheduled routine maintenance”
the NSA, America's largest intelligence agency

The same Tuesday, the NSA's classified network went offline. The agency blamed "scheduled routine maintenance".

See the story →

In the news · 18 Nov 2026

“zero evidence”
Marco Rubio, US secretary of state

Marco Rubio, the US secretary of state, said there was "zero evidence" that the week of outages was an attack.

See the story →

The idea

most outages start with a change

Engineers who keep big systems running call this work change management: making changes so they break as little as possible. It starts from one plain fact: most outages start with a change.

Seen it before

Google's site reliability engineers, the people who keep its services running, found that roughly 70% of outages are due to changes in a live system.

A change is anything new in a working system: an update, a new router, a fix, a new setting. Each one meets the real system for the first time.

Quick check

An airline's booking site fails at 9am. Overnight it installed new software, a hacking group was in the news that week, and bookings were normal. What does change management look at first?

Tap the answer you think is right.

Tap any other answer to see why.

Why does a tested change still fail?

Swipe up to start ↑Scroll down to start ↓

A change is tested before it goes out. But a test is never the real thing, with every real user, old part and odd case.

In the news · 18 Nov 2026

freeway rides paused in May

Waymo runs driverless taxis in 15 US cities. In May it stopped its own freeway rides to fix how its cars handled roadworks.

See the story →

In the news · 18 Nov 2026

In July it went back on freeways with a software update for roadworks.

See the story →

In the news · 18 Nov 2026

On 5 November, 3 Waymo cars drove into a closed roadworks zone on US Route 101 near Burlingame, in California.

See the story →

In the news · 18 Nov 2026

“imminent hazard to public safety”
California's motor vehicles department

California's motor vehicles department said the cars pose an "imminent hazard to public safety" and banned them from its freeways. Waymo stopped its freeway rides everywhere else in the US too.

See the story →

In the news · 18 Nov 2026

Quality engineers call a fix for a fault's cause a corrective action, and judge it by whether the fault comes back. Waymo's July update was one, and in November its cars drove into roadworks again.

See the story →

In the news · 18 Nov 2026

“effective corrective actions”
California's motor vehicles department

A fix only counts once the fault stops coming back. California's ban has no end date, and Waymo must first show the state "effective corrective actions".

See the story →

There is a second reason. A change that reaches everyone at once can break for everyone at once.

In the news · 18 Nov 2026

one update, every user

Apple's internal update reached iCloud for all users at once, so one bad change hit everyone.

See the story →

Seen it before · 2024

one faulty update, 19 July 2024

On 19 July 2024, CrowdStrike, a security company, sent a faulty update to the computers running its software.

Microsoft estimated that it affected 8.5 million Windows devices, less than 1% of all Windows machines.

CrowdStrike pulled the update back 78 minutes later. The computers hit were those switched on and sent it in that time.

Quick check

A bank's app update locks out all 10 million of its customers within an hour. What best explains why so many were hit?

Tap the answer you think is right.

Tap any other answer to see why.

How careful operators change things

Swipe up to start ↑Scroll down to start ↓

The habits

Google's engineers name three habits for changes: roll them out in steps, spot problems fast, and roll back safely.

Rolling out in steps means a change reaches a small group first, then more. Engineers call the first small group a canary.

a small early warning

Coal miners once carried canaries underground. The birds fell ill from poison gas before the miners noticed it, which warned the miners to get out.

Seen it before · 2024

“starting with a canary deployment”
CrowdStrike

After July 2024, CrowdStrike said that type of update would now go out in stages, "starting with a canary deployment".

Rolling back means undoing a change and going back to the version that worked.

Spotting problems fast means watching the first group closely, so a bad change is caught early. CrowdStrike's faulty update ran for 78 minutes before it was pulled back.

In the news · 18 Nov 2026

new router installs paused

After Friday's halt, the FAA paused putting in new network routers.

See the story →

Engineers call that a change freeze: no new changes until the last one is understood.

In the news · 18 Nov 2026

“identifying the device that failed does not establish why it failed”
US senators, to the FAA

US senators wrote to the FAA that "identifying the device that failed does not establish why it failed".

See the story →

In the news · 18 Nov 2026

root cause not yet found

The FAA has not found the root cause yet, so nobody can say the other new routers are safe.

See the story →

The spare

A fourth habit is a spare. Engineers call it redundancy: a spare system that takes over when the main one fails.

In the news · 18 Nov 2026

backup centres carry the work

While the NSA's secret network was down, its backup centres around the US were expected to carry the agency's work.

See the story →

Quick check

A game studio's new update crashes the game, but only for the first 1% of players who got it. Which habit kept the damage small?

Tap the answer you think is right.

Tap any other answer to see why.

Patch now, or change carefully?

Swipe up to start ↑Scroll down to start ↓

In the news · 18 Nov 2026

a critical flaw in Java

On Tuesday researchers made public a critical flaw in Java that can let attackers take over servers without a password.

See the story →

In the news · 18 Nov 2026

CISA: patch now

Oracle shipped patched Java builds the same day. CISA, the US cyber-defence agency, urged organisations to patch immediately.

See the story →

Why the hurry?

Every day a fix sits uninstalled, attackers can use the hole. Security engineers call that time the patch gap.

In the news · 18 Nov 2026

ConnectWise fixed a flaw in ScreenConnect, a tool IT firms use to reach their customers' computers remotely, on 8 September. Attacks on servers still missing the fix ran into November.

See the story →

In the news · 18 Nov 2026

Huntress, a security firm, counted 688,000 attack attempts on 42 servers.

See the story →

In the news · 18 Nov 2026

Fixes are coming faster because AI finds flaws faster. A researcher found the Java flaw in about an hour using OpenAI's Codex, an AI coding assistant.

See the story →

In the news · 18 Nov 2026

PostgreSQL, a free database used by many big firms, patched 52 security holes in one quarterly release, a record. Some of the reports came from AI scanning tools.

See the story →

In the news · 18 Nov 2026

Anthropic will give software makers 30 days, not 90, to fix serious flaws its AI finds before it publishes them. So makers get less time to build and test each fix.

See the story →

In the news · 18 Nov 2026

everything patched at once

Mark Warner, a US senator, says firms now patch everything at once. That strains systems never built for that pace, he says.

See the story →

In the news · 18 Nov 2026

upgrades and patches all at once

Jay Clayton, the US director of national intelligence, blamed the week's outages on organisations upgrading and patching old systems at the same time.

See the story →

Two sides

Wait to patch, and attackers get longer to use the hole. Patch everything at once, and the fixes themselves can knock the system over.

In the news · 18 Nov 2026

hit whatever the cause

People cut off from a bank, a flight or work email were hit whatever the cause, says Tal Kollender of Remedio, a security company.

See the story →

Quick check

A hospital waits one week to install a security fix, so it can try it on a few machines first. What is it trading?

Tap the answer you think is right.

Tap any other answer to see why.

What to carry

Swipe up to start ↑Scroll down to start ↓

Carry this · 1 of 3

Most outages start with a change somebody made, not with an attacker.

Carry this · 2 of 3

A fix that passes its tests can still fail on the real system, and a change that reaches everyone at once fails for everyone at once.

Carry this · 3 of 3

As AI finds flaws faster, fixes come faster too, and every fix is a change that can break something.

That's the course.

Why do the biggest outages start with a fix? 4 parts, about 14 minutes.