Outage disrupted jails, hospitals, schools and police reporting

Microsoft’s initial report, issued Monday, identified a months-old bug malfunctioning connections between internal systems among the factors that led to the blackout. The company said each factor might have been resolved on its own but, together, produced an outage that was harder to mitigate.

A latent bug or defect from July caused Microsoft’s servers to “become overwhelmed and unresponsive”, Charles Lamanna, the executive vice-president of Copilot, Agents, and Platform at Microsoft, said in a post on X on Monday.

Lamanna wrote that Microsoft normally has two networking gateways available — entry and exit points that join separate systems and allow data to pass between them. One of them was down for Microsoft Teams, which overloaded the primary gateway. Over 14% of the company’s server capacity was out of service because of an unrelated issue from the previous day.

“Taken together, the increased load and the reduced server capacity, caused the network gateway to encounter the latent bug from July,” Lamanna wrote. “This caused requests to build up until servers ultimately stopped responding. This occurred during the peak window of traffic in the United States.”

US users were the first to report trouble on Thursday, saying they could not get into their Outlook email accounts. By the next day, millions of people worldwide said they had lost access not only to their email but to other Office 365 services as well. The failures reached essential services: jails were unable to release prisoners, hospitals fell back on paper records, schools lost touch with parents, and police departments could not receive crime reports. Microsoft said it had resolved the issue and restored service by Saturday, and it issued an apology.

Lamanna said the company had trouble mitigating the bug because it originated months earlier, so undoing it would not be as simple as rolling back a recent change.

“Recovery took longer than usual because as servers returned to service, they were immediately overwhelmed by waiting traffic and became unresponsive again,” he wrote. “Our normally automated recovery processes were not designed to restore service from this broad unresponsive state. Additionally, since the main code defect was a latent bug from several months previous, our normal rollback, flighting, and feature flags were unable to quickly revert the change.”

The company did not attribute the blackout to a cybersecurity breach, and Lamanna said Microsoft plans to publish a final report that will include more information.

Analysts say Microsoft’s global outage highlights the need for companies to examine how much they rely on a single operating system. Matt Hull, vice-president of cyber intelligence and response at NCC Group, said: “This is a prime example of the importance of not only cyber resilience, but business resilience.” He added: “Reports that hospitals, schools and other essential services are affected demonstrate the wide impact that a system outage can have, regardless of the cause.”