字号·· | 护眼
theregister

微软将大规模电子邮件故障归咎于7月修复中引入的软件缺陷Microsoft blames monster email outage on software defect introduced in July fix

点「原文对照」整页切到原文,或双击某段只看那段的原文。

微软将上周的重大故障归咎于其在7月作出的一项更改,该更改潜伏在网络网关服务器中,直到负载激增和容量下降触发了这一缺陷,让整个Microsoft 365世界放假一天,无法收发电子邮件。根据微软于11月15日发布的初步事故后报告,该缺陷于2026年7月引入。微软表示,当时该更新看起来没有问题,它通过了负载测试,并在部署时没有产生任何影响。

Microsoft has blamed last week's epic outage on a change it made in July that lurked in the network gateway servers until a surge in load and a drop in capacity triggered the defect, giving the Microsoft 365 world a day off from email. According to a preliminary post incident report published by Microsoft on November 15, the defect was introduced in July 2026. The update looked fine at the time, Microsoft says, having passed load testing and been deployed without any impact.

然而,测试还不够充分。微软的报告指出:“我们的负载测试倾向于关注核心较少的较弱硬件。”该缺陷导致“内部Windows进程内部发生锁争用,从而耗尽了机器的资源(CPU和内存)。”“在拥有更多核心和较新硬件的机器上,争用问题会更加恶化。”包含该缺陷的修复程序是专门为Microsoft 365网络网关服务器准备的。

However, the testing wasn't good enough. Microsoft's report states: "Our load testing tended to focus on weaker hardware with fewer cores." The defect caused "lock contention inside an internal Windows process which caused the machine's resources (CPU & memory) to be exhausted." "The contention issue is worsened on machines with more cores and newer hardware." The fix containing the defect was prepared specifically for Microsoft 365 network gateway servers.

然后两件事同时发生了。Teams故障转移到了支持Exchange Online的“主”网关,造成了负载激增,这本身并不是特别不寻常。与此同时,网络网关近14%的服务器容量处于停机状态,这本身也不是问题:这些服务器正在恢复中,据微软称,“服务具有充足的容量余量。”这两者结合在一起,将网关推向了缺陷,随后陷入了混乱。

Then two things happened at once. Teams failed over to the "primary" gateway that supports Exchange Online, creating a surge in load that was not, in itself, particularly unusual. At the same time, almost 14 percent of server capacity for the networking gateway was out of service, which wasn't a problem by itself either: it was on its way to being restored and, according to Microsoft, "the service had sufficient capacity headroom." Together, they pushed the gateway into the defect and chaos ensued.

这是一个典型的级联故障场景。当服务器失去连接时,流量会被路由到下一个健康的端点。而该端点遇到了同样的问题。因此流量被再次重新路由,而且……你可以看出事情会发展成什么样。在两个小时内,微软的整个服务器集群都遇到了该缺陷。

It was a classic cascading failure scenario. When a server loses connectivity, traffic gets routed to the next healthy endpoint. Which encountered the same problem. So traffic was rerouted again and… you can see where this was going. Within two hours, the defect was encountered across Microsoft's fleet.

报告指出:“在最低谷时,超过80%的服务器无响应。”那么,该怎么做?将服务器重新投入使用并没有用——它们立即不堪重负并再次瘫痪。“我们通常自动化的恢复过程并非旨在从这种广泛的无响应状态中恢复服务。”直接回滚也是不可能的,因为该缺陷已经存在数月之久。相反,工程师们开发了针对该补丁的补丁,或者用微软的话说,“一个将代码路径恢复到Windows中先前已知良好状态的更新。”该补丁被应用于各组服务器,服务逐渐恢复。

"At its lowest," the report states, "more than 80 percent of servers were unresponsive." So, what to do? Bringing servers back to service didn't help – they were immediately overwhelmed and fell over again. "Our normally automated recovery processes were not designed to restore service from this broad unresponsive state." A straightforward rollback was also not possible because the defect was months old. Instead, engineers developed patch for the patch or, in Microsoft's words, "an update that returned the code path to the previously known good state in Windows."

时间表显示了客户所经历的痛苦,以及微软在雷德蒙德试图弄清楚为什么Exchange Online和其他云服务表现如此糟糕时的手足无措。事实证明是导致故障因素之一的Teams故障转移发生在协调世界时11月12日01点07分。白天,负载开始上升,直到协调世界时16点03分,微软的连接异常检测监视器发现连接问题有所增加。

The patch was applied to groups of servers and service was gradually restored. The timings show the pain customers experienced and the floundering in Redmond as Microsoft tried to work out why Exchange Online and other cloud services were so poorly. The Teams failover, which turned out to be a factor in the outage, happened at 0107 UTC on November 12.

在协调世界时16点33分,第一个支持案例进来了。不难想象工程师们感到的压力,在接下来的15个小时里,他们尝试的任何方法都没有解决问题。所有正在进行以及计划中的部署和配置更改都被暂停,同时团队试图恢复服务。增加了额外的容量。重新路由了流量。然而服务器却不断瘫痪。

During the day, the load began to ramp up until, at 1603 UTC, Microsoft's connectivity anomaly detection monitor identified an increase in connectivity issues. At 1633 UTC, the first support case came in. It is not difficult to imagine the stress engineers felt as, over the next 15 hours, nothing they tried resolved the issue. All ongoing and planned deployments and configuration changes were paused while the team attempted to resurrect the service. Extra capacity was added. Traffic was rerouted.

报告称:“那些能够做到的服务——比如Teams、OneDrive、SharePoint和Office应用——退出了有缺陷的网络网关基础设施。”“但并非所有服务,比如Exchange Online,都能出于规模原因退出这个网络网关。”因此,Exchange Online(或者从许多用户的角度来看是Outlook)中断的时间很长。到协调世界时11月13日00点30分,微软已将连接协商确定为访问失败的罪魁祸首,并得出结论认为相互竞争的进程阻碍了连接的处理。

Yet servers kept falling over. "Those services that were able – like Teams, OneDrive, SharePoint, and Office apps – failed out of the faulty network gateway infrastructure," the report says. "But not all services, like Exchange Online, can fail out of this network gateway for scale reasons." Hence the length of time for which Exchange Online (or, from the perspective of many users, Outlook) was out of action.

然而,直到协调世界时07点20分,“我们才确定了一个缓解措施并应用了该缓解措施——一个还原了有缺陷更改的更新。”在协调世界时16点10分,微软确认情况再次恢复正常,并在11月14日04点15分发布了其最终的客户更新。

By 0030 UTC on November 13, Microsoft had identified connection negotiation as the main culprit for the access failures and concluded that competing processes were preventing connections from being handled. However, it took until 0720 UTC before "We identified a mitigation and applied the mitigation – an update that reverted the defective change."

最终的事故后报告定于11月20日发布。然而,这份初步报告对于平复紧张的神经几乎起不到什么作用。三个月前应用的一个更新竟然导致了全球范围内的Exchange Online(和其他服务)中断?负载测试没有反映微软控制下的场景?那些狂热地工作以恢复服务的工程师理应受到表扬,并且希望他们在雷德蒙德再也不用自己买饮料了(或者至少能逃过下一轮裁员)。然而,导致带有潜伏缺陷的更新进入生产环境的流程值得进行深入而严厉的审视。®

At 1610 UTC, Microsoft confirmed that things were once again looking healthy and, at 0415 on November 14, issued its final customer update. A final post incident report is due on November 20. However, the preliminary report will do little to calm jangling nerves. An update applied three months ago managed to cause Exchange Online (and other services) disruption on a global scale? With load testing that didn't reflect scenarios under Microsoft's control? The engineers who worked feverishly to restore services deserve commendation and will hopefully never have to buy their own beverages in Redmond again (or at least escape the next round of firings). However, the processes that led to an update with a latent defect reaching production deserve a long, hard look. ®

阅读媒体原文 ↗