01 · Briefing · what happened

Cloud outages are moving up to the control plane - where your backups can't reach

Information Technology 1 min 3 sources

A growing share of cloud failures now hit the management layer, not the servers. When that layer wobbles, redundancy below it doesn't help - and a UK bank's ten-day shutdown showed the choice it forces.

10+ days

CAF Bank offline

online service withdrawn after suspected fraud

~$600bn

cloud-giant capex in flight

ever more workloads on a few control planes

At a glance

A new analysis says more cloud outages now come from the control plane - the provider's management layer - not from failed servers.
When that layer wobbles, teams can't make changes or trust the system's state, even while the hardware still runs.
The Uptime Institute reports the same shift: fewer isolated hardware faults, more management-layer failures.
Redundancy below the control plane doesn't save you - two regions sharing one identity system share one weak point.
CAF Bank took online banking offline for over ten days rather than run on a system it couldn't trust.
The choice underneath: keep serving on shaky state, or hold still to stay correct.

Forces in play

Control-plane risk Building↑

outages shifting to the management layer

Redundancy's reach Steady→

regions still share one control layer

Provider concentration High↑

more workloads on fewer platforms

Recovery realism Easing↑

architects told to plan for degraded control

In play Cloud architects — learning that geographic spread is not operational independence The big providers — the shared control planes many regions depend on at once CAF Bank — chose to go dark rather than serve untrusted state

Where this points

Watch whether the next headline cloud outage traces to the control plane rather than a server or region - that is the tell this shift is real, not a one-off.

Full briefing

For years, cloud teams built resilience the same way: spread workloads across regions, replicate the data, write a failover plan. It looks mature on paper. A new analysis of recent incidents says the failures increasingly come from somewhere those defences don’t cover - the control plane, the provider’s own management layer of APIs, identity, and automation that lets you make changes and trust what the system is doing.[1]

When that layer wobbles, the servers can still be running while teams “could not make timely changes, trigger the recovery actions they expected, or trust the environment’s state in real time.”[1] The Uptime Institute has flagged the same shift - more outages tied to control-plane failures than to isolated hardware faults.[1] The blast radius is wider than most plans assume: two regions that both lean on the same identity system and the same orchestration share a single point of failure above the redundancy.[1]

A live example of the underlying choice: CAF Bank pulled its online banking offline for more than ten days after spotting “attempted fraudulent activity,” and warned access would stay intermittent - it “may need to limit the amount of traffic to the website.”[2] Faced with a system it couldn’t fully trust, the bank chose to stop serving rather than keep serving on shaky ground.

That is the trade under the hood of every distributed system. When the coordinating layer splits or stalls, you can keep answering requests (and risk stale or wrong state) or you can hold still to stay correct (and go dark). The concentration of ever more workloads onto a handful of providers’ control planes only raises the stakes of that one choice.[3]

02 · Lesson · why it matters

You can't have both - you can only choose what breaks

When a shared system loses touch with itself, it must pick: keep answering and risk being wrong, or stay correct and go dark. There's no third door.

How it works

A network or coordination layer splits under stress
The system can't both stay correct and keep answering
It must sacrifice one guarantee
The guarantee it drops decides exactly what breaks

The twist

You never really 'have' both consistency and availability - you've just picked one to sacrifice, and quiet conditions hid the pick until stress revealed it.

Where you've seen this

ATMs offline

a cash machine that can't reach the bank either refuses you or risks a double withdrawal

Shared documents

edit offline and the app must merge conflicts later or lock you out now

A team out of contact

act alone on old orders, or wait for word and stall

The catch

The choice only bites during a partition - most of the time you can have both, which is exactly why teams forget they ever chose.

Full lesson

The rule hiding under the cloud

Computer scientists have a name for the trap behind today’s control-plane outages. It’s called the CAP theorem, and it’s less a formula than a hard fact about shared systems.

Picture a service running in two places at once, so one can cover if the other fails. Between them runs a wire - the coordination that keeps both copies agreeing on the truth. Now the wire is cut, or just too slow to trust. The two halves can no longer check with each other.

At that instant, each half faces a choice. It can keep answering requests using what it last knew - fast, available, but maybe wrong. Or it can refuse to answer until it can confirm the truth - correct, but silent. It cannot do both. That is the whole theorem: when the network splits, consistency and availability stop being friends and become a trade.

Why the redundancy didn’t save them

This is why the enterprises in this week’s story were surprised. They did everything the checklist said - many regions, replicated data, documented failover. Their mistake was believing more copies meant more safety.

But copies don’t help when the thing that broke is the layer above them. Both regions ask the same management layer the same questions: what’s the current state, where do I send traffic, am I allowed to do this. If that layer can’t answer clearly, both regions are stuck in the same fog at once. The redundancy is real. It’s just below the fault, not above it.

Geographic separation, it turns out, is not the same as operational independence. Two data centres a continent apart can still be one system if they share one brain.

The choice defines what breaks

Here’s the part worth carrying past today. A distributed system never really “has” consistency and availability both. It has picked one to give up when trouble comes - and quiet days simply hide the pick.

A bank that takes its site offline for ten days rather than run on state it can’t trust has chosen consistency: better to serve no one than serve wrong numbers. A shopping cart that lets you keep adding items during an outage has chosen availability. It sorts out the double-charges later: better to keep the doors open and clean up after. Neither is right in the abstract. Each is a decision about which failure you can live with.

The danger isn’t making the trade. It’s not knowing you made it. A system designed as though it will always be reachable and always correct has quietly chosen for you. You find out which, at the worst moment, when the wire goes down.

Everywhere two things must agree from far apart

Once you see the shape, it’s not really about servers. It’s about any two parts of a whole that must agree while being apart.

An ATM cut off from the bank either refuses your cash or risks paying you twice. A ship’s crew out of radio contact must act on last week’s orders or wait and lose the moment. Two managers who can’t reach each other during a crisis either move on their own reading or stall for a decision that isn’t coming. In each, the same fork: act on what you have and risk being wrong, or hold for certainty and risk being late.

Humans reach for “we’ll just have both” - full speed and full safety, autonomy and alignment, move fast and never break. The theorem is a quiet correction. Under stress, when the parts lose touch, both is not on the menu. You will sacrifice one. The only real choice is whether you chose it on a calm day, with eyes open, or let the outage choose it for you.

What the whole looks like from inside

The people who built these systems are not careless. They’re standing inside a web too large for any one seat to hold. The architect who spread workloads across regions saw redundancy and truly had it - for the failures they could picture. The layer they leaned on was invisible precisely because it worked, every day, until it didn’t.

That is the humbling part. You can build the whole thing right and still be caught, because “right” was measured against the failures you imagined, and the one that came was above them. Seeing this doesn’t hand you a fix. It hands you a smaller, truer question to carry into any shared system you depend on - a payment network, a supply chain, a team. When this loses touch with itself, which promise does it keep, and which does it quietly drop? You may not like the answer. But it’s better to know it before the wire goes down.

03 · Lab · your turn

The Wire Just Cut

Rehearse the CAP choice: when a system loses touch with itself, keep serving on shaky state or hold still to stay correct - and feel that the right call depends on the cost of being wrong.

04 · Hope · carry this

The engineers caught out today write tomorrow's failover plan - we get better at holding systems together by studying, without blame, exactly where the last ones came apart.