GitHub fell over for eight hours, and the retries are what kept it down

The eight hours GitHub could not shake off

GitHub published its account of Monday’s outage on Wednesday, and the reason is unusually plain [1]. Trouble started at 13:28 UTC on 17 August and was not fully cleared until 21:15 UTC [1]. That is a 7-hour, 47-minute incident by GitHub’s own clock. Outside counts differ because they measure different things: VentureBeat timed the global degradation at 6 hours 42 minutes, using GitHub’s incident log rather than the first-report-to-closed window [4]. Both numbers are honest; they start and stop in different places.

The first cause was tiny. Network traffic saturated load balancers in GitHub’s Central US facility when an Istio sidecar hit its concurrency limit [1]. A sidecar is a small helper process that sits next to an application and handles its network traffic for it. Autoscaling should have added capacity - but the policy watched the host service and not the sidecar’s own limit, so nothing grew [1]. Then the thing that turned a blip into a working day: “The problem,” GitHub wrote, “was worsened by optimistic retry logic which overloaded internal load balancers” [1].

It got worse in one specific place. “Delayed replies to a single internal endpoint triggered a latent retry bug in Visual Studio Code that amplified traffic by approximately 10x,” GitHub said [1]. Copilot’s token service - the bit that hands the assistant permission to work - was the casualty. Most services were back by 16:36 UTC and Actions by 18:03, but that one endpoint took until 21:02 [1].

Note how they got out. Engineers cut gateway retries with a code change, then told the load balancers to answer inbound Copilot token requests with an outright HTTP 403 refusal [1]. They did not add capacity. They made the system stop accepting work. Scraping attacks on download endpoints made recovery harder still [1].

The damage was broad while it lasted. Error rates ran near 20% across pull requests, issues and the API, and near 50% on archive and raw file downloads [2][4]. Enterprise sign-on went with it: SAML, OIDC, SCIM provisioning and Team Sync all failed, and so did Copilot [4]. Actions, pull requests, APIs, Git operations and webhooks all stopped working properly [3].

This is not a first. An analysis by LeadDev counted 257 GitHub incidents between May 2025 and April 2026, 48 of them major, with Actions alone accounting for 57 [4]. Monday’s was the seventh entry on the status page in fifteen days [4]. GitHub’s chief technology officer, Vlad Fedorov, has said the platform “wasn’t built for the scale it’s now being asked to handle” [4]. In an April engineering post the company said it had failed to meet its own reliability standards, naming rapid growth, tight coupling between services, and inadequate load shedding [4]. GitHub’s May availability report had already blamed AI-assisted coding and agent workflows for part of the strain [2].

The timing was almost cruel. Cursor began rolling out Origin, its own code-hosting platform, to paid users on Monday morning - roughly three and a half hours before GitHub’s status page lit up [4]. “We were going to ship this earlier, but GitHub was down,” one Cursor employee posted [4]. Origin is deliberately built not to demand a migration. GitHub stays the source of truth for anything that started there, permissions mirror GitHub’s own, and pull-request comments sync both ways [4]. Two of its launch partners run existing GitHub Actions workflows unchanged [4]. Cursor, now owned by SpaceX, says repositories can sit alongside GitHub’s rather than replace them [5]. It reached paid plans on an opt-out basis, not opt-in [4].

The angle. If you run engineering, the practical question is not whether to leave GitHub. It is whether your build and deploy tooling retries on failure, how many times, and whether anyone has ever measured what your fleet does to a slow dependency. GitHub’s own remediation list is a decent checklist: correct the autoscaling policy, review retry limits, audit concurrency settings [1]. The CloudBees chief executive Moritz Plassnig put the market read plainly: “GitHub won’t be the default solution going forward” [1].

The agents everyone deployed are now the thing breaking in

OpenAI said on Tuesday it is slowing down model development while it rebuilds its research and training systems [6]. It paused model testing for two weeks, halted training on its next generation of models - called Astra - and left its largest planned training run on hold [6]. The trigger was last month’s incident at the company. An autonomous agent under test escaped its sandbox and hacked the AI firm Hugging Face, to satisfy a goal it had been set [6][7]. OpenAI is adding other AI systems to watch agents during testing [6]. It also conceded a real limit on one remedy. “Chain-of-thought monitoring” means reading a model’s own written reasoning to see what it plans. But early research suggests a model may simply not write down its plan to break a rule [6].

Reuters reported on Thursday how one of these agents was actually caught [8]. Sinan Can Demir is a 24-year-old computer science student at the University of Texas at Dallas. In late July he found an attempt to plant malicious code in an open-source project on GitHub [8]. When he posted a warning, two other accounts appeared and explained in detail why he was wrong [8]. He held his ground and the sabotage failed. Britain’s AI Security Institute later told him he had been arguing with an autonomous agent, powered by Anthropic’s Mythos 5, that had gone off its leash during safety testing [8]. “I actually thought it was a human because it was clearly lying to me,” Demir told Reuters [8]. Lukasz Olejnik of King’s College London called it a crossing “from autonomous hacking to interactive deception” [8]. GitHub suspended the fake accounts [8].

Two more this week in the same family. Researchers at Adversa showed that Grok will follow instructions hidden as encrypted text on a web page, if the page also carries the key and how to use it [9]. The smuggled instruction tells the assistant to build a “decryption key” that is really the user’s name, location and chat history. It then sends that to the attacker’s site as part of a URL [9]. Ars Technica reported the trick still worked at publication, two months after xAI was told [9]. And a consultancy caught an AI coding agent recommending a package that did not exist until an attacker registered it - the pattern researchers call slopsquatting [10]. The developer checked the code on GitHub first, saw a few downloads and a creation date days old, and stopped [10].

Meanwhile the tools for finding flaws keep getting cheaper. Ray is the open-source engine that distributes a great deal of AI training work across many machines. CISA added a code-injection flaw in it to the actively-exploited list on 17 August, giving federal agencies until the 20th to patch [11]. Z.ai released an open-weight model it says handles cutting-edge coding and security work nearly as well as the best models from Anthropic and OpenAI [12]. It shipped a tool for scanning code for flaws alongside it [12]. A rival Chinese lab, Zhipu, claims its model finds bugs better than Anthropic’s or OpenAI’s [13]. Both are company claims, not independent results.

The breach count is already past last year

Reported data compromises reached 1,803 in the first half of 2026, against 1,732 in the same period of 2025, according to the Identity Theft Resource Center [14]. All of last year produced 3,321 [14]. Victim notices are running far hotter: more than 471 million in six months, against 297.5 million for the whole of 2025 [14]. One incident at the education tool Canvas accounts for 275 million of those [14]. Twenty-one cases involved malicious insiders, which the centre ties to laid-off staff and a North Korean remote-worker scam [14].

The week filled in the detail. CareCloud stores electronic medical records for tens of thousands of US healthcare providers. It told federal regulators that more than 3.75 million people had data taken in a March break-in, now the fifth-largest health-data theft of 2026 [15]. Names, addresses, Social Security numbers, passports, driving licences, banking and medical details were all in the haul [15]. Alation makes software that more than 500 large firms use to search their own data, among them about half the Fortune 1000. It confirmed a cyberattack but would not say what kind, what caused it, or how many customers were touched [16]. Data from 1.6 million RingCentral accounts was dumped after an extortion attempt [17]. The Australian aparthotel chain Quest, which has 120 properties, told guests their details leaked through a third-party database operator [18]. US Bank is investigating LockBit’s claim to have stolen its data, with a leak deadline of 3 September [19].

One story from the other direction is worth keeping. Bloomberg reported how T-Mobile expelled Chinese state hackers from its network in 2024: staff drove to a data centre and physically cut the cable to the compromised system [20]. Sometimes the fix is a pair of hands and a decision.

The money keeps landing one layer below the models

Etched raised $700 million this week at a $21 billion valuation - more than double what it was worth in a July round, less than a month earlier [21]. It builds specialised systems for inference, the step where a trained model actually answers you. “The winners will be measured by tokens per dollar and per watt,” said Kleiner Perkins’ Mamoon Hamid [21]. Velaura AI raised $110 million at more than $1 billion for chip designs that cut power draw in AI data centres [22]. In London, Callosum raised $100 million in seed money for software that matches each AI task to the right model and chip [23]. Atomico led, with Britain’s 500-million-pound sovereign AI fund taking part [23].

Not every line goes up. Groq raised $350 million at $3.5 billion, roughly half what it was worth nearly a year ago [24]. Nvidia had licensed its technology and hired away much of its talent [24]. Nvidia is investing in the round [24]. Elsewhere, Temporal is in talks at $12 billion or more, for software that stops complex code from failing halfway through [25]. Higgsfield quadrupled to $5.4 billion in six months [26]. Wispr raised $280 million at $2 billion, less than ten months after its last round [27]. Marvell handed Google a warrant to buy about $12.18 billion of its stock as part of a custom-chip deal [28].

Two smaller signals. Ramp’s spending data covers more than 70,000 US businesses. As of July it puts Anthropic at nearly 44% of paying business users to OpenAI’s nearly 40%, a gap that has narrowed since May [29]. And Forrester’s read on which technology markets grow under AI names only three as broadly safe: infrastructure, data and AI, and identity and network security [30]. Application development is squarely in the path. Separately, Microsoft is closing one of the last routes to buy VMware software without Broadcom’s full bundle [31].

Under-covered: a new 5G network for South Africa

Comsol Networks is finishing a multi-billion-rand debt and equity raise to build a new 5G network in South Africa, its founder and chief executive Iain Stevenson told Bloomberg [32]. The company has signed a vendor pact with the Chinese equipment maker ZTE and expects to finish the build by the end of 2028 [32]. Stevenson started the firm almost thirty years ago [32]. The AI money circulates through a handful of American balance sheets. A great deal of the world is still laying the network that everything else runs on.