Microsoft-365-Recovery

Microsoft blames earlier network upgrade for 365 outage as incident stays open

Featured News

Microsoft has apologized and reported broad recovery, but its account of the cause differs from what it told customers on Thursday, and its full review has no published date.

Microsoft has attributed this week’s Microsoft 365 outage to a problem introduced during an earlier network capacity upgrade, apologizing to customers while keeping the incident open more than 35 hours after Exchange Online first failed.

“We are deeply sorry for the disruption this incident has caused our customers and partners,” Microsoft said in a post on its corporate X account on Friday.

The company said the incident “was introduced by an issue during an earlier network capacity upgrade that surfaced following a recent network change.” It called it “a technical issue not related to cyber or AI / agents” and said the underlying issue had been addressed. Microsoft did not say what the upgrade involved, when it was carried out, or which network change exposed the problem.

The outage began at 16:17 UTC on Nov. 12 in Exchange Online. It spread to Teams, SharePoint Online, Microsoft 365 Copilot, Defender XDR, Defender for Office 365, Intune, Planner and To Do. Downdetector recorded disruptions in 74 countries and territories.

John Joyner, senior director of technology at Corsica Technologies, said the outage was principally an Exchange Online failure.

“Because services like Microsoft Teams, Copilot, SharePoint, and OneDrive rely on Exchange Online infrastructure for calendars, meetings, and work data access, they experienced widespread secondary degradation,” Joyner said.

A changing account

The explanation differs from what Microsoft told customers during the incident. On Thursday afternoon, the Microsoft 365 Status account said that “We’ve not identified any recent change correlated with the start of impact.” Microsoft had already paused all deployments and configuration changes while it reviewed recent changes.

Later that night, Microsoft said in a service health update that it had “begun reverting a recent network configuration change that correlates with the start of impact.” The same update said a process on affected servers “responsible for establishing secure connections becomes overwhelmed, causing the servers to stop processing requests.”

In its Friday statement, Microsoft also addressed how long recovery took.

“Given the scale of demand, we restored services gradually, validating stability at each stage. While this took longer, it helped ensure a stable restoration,” the company said.

Australian Home Affairs Minister Tony Burke gave a similar account at a news conference in Sydney on Friday, before Microsoft’s statement. He had been briefed by the company.

“They believe they are dealing in some ways similar to what we saw with respect to CrowdStrike where there is an update that has gone awry,” Burke said.

Recovery, but not resolution

Microsoft reported recovery on Friday across North America, much of Europe and Asia Pacific. It later said email queues were “significantly reduced and approaching normal levels.” To keep delayed mail from failing, it extended the email timeout to 48 hours.

The incident remained open as of Saturday.

“Due to the severity of impact this incident caused, we’re monitoring for an extended period to ensure no additional mitigative actions are required,” Microsoft said in its latest update.

Kate Wood, AVP of research development at Info-Tech Research Group, said the lag between recovery and resolution leaves IT leaders with a decision to make.

“CIOs need a clear basis for telling the business it can return to normal operations,” Wood said. “That means knowing which critical processes have been tested, where security coverage still needs checking, and which teams must continue using workarounds.”

Microsoft said it will “provide a full post-incident report once our review is complete,” and gave no date. Earlier service health updates said the report would include the root cause and “defense-in-depth repair items.”

Workarounds and security

During the incident, Microsoft’s service health advisory warned that “Safe Links may be unable to verify URLs, which may impact users clicking links in Outlook and Teams.” Joyner said the longer an outage runs, the more risk shifts to employee behavior.

“With longer disruptions, security increasingly depends on how people work around unavailable services,” he said. “Employees may move sensitive files to personal accounts by using personal email accounts, while responders lose access to email threads they need to coordinate an investigation.”

Joyner said IT teams should activate an approved alternative communication channel and tell employees which workarounds are permitted. He also recommended arranging out-of-band channels, such as SMS broadcast, in advance. Adding that contingency information to regular security training can help prevent what he called “shadow I.T. activation,” he said.

This is the fourth significant Microsoft 365 disruption this year. It follows a roughly 10-hour outage in January, a July outage that Microsoft attributed to a bug in automated network maintenance, and an Exchange Online failure in late August.

Microsoft has not said whether affected customers will receive service credits.

Leave a Reply

Your email address will not be published. Required fields are marked *