字号·· | 护眼
印度斯坦时报

微软为全球Outlook服务中断道歉,归咎于隐藏的服务器缺陷、流量增加和容量减少Microsoft apologises for global Outlook outage, blames hidden server defect, increased traffic and reduced capacity

点「原文对照」整页切到原文,或双击某段只看那段的原文。

微软已就上周在多个国家导致其电子邮件服务中断的全球Outlook故障道歉。该公司还解释了导致故障的三个主要原因,该故障影响用户近两个工作日。

Microsoft has apologised for the global Outlook outage that disrupted its email services across several countries last week. The company has also explained the three main reasons behind the outage, which affected users for nearly two working days.

微软表示,故障是由其服务器中的隐藏软件缺陷、网络流量突然增加以及服务器容量下降共同造成的。这三个问题同时发生,最终导致服务器停止响应。

Microsoft said the outage was caused by a combination of a hidden software defect in its servers, a sudden increase in network traffic and reduced server capacity. These three problems occurred together and eventually caused servers to stop responding.

微软Copilot、代理和平台执行副总裁查尔斯·拉曼纳周一在X上发布的一篇详细帖子中解释了故障背后的原因。他说,这些问题影响了支持多个微软365服务的网络网关。微软Outlook故障:隐藏服务器缺陷导致电子邮件中断第一个问题是隐藏的软件错误。拉曼纳说,当服务器处理某些类型的互联网流量时,该错误导致微软服务器过载并停止工作。

Charles Lamanna, Microsoft's Executive Vice President of Copilot, Agents and Platform, explained the reasons behind the outage in a detailed post on X on Monday. He said the problems affected the networking gateway that supports several Microsoft 365 services. Microsoft Outlook outage: Hidden server defect caused email disruptionThe first problem was a hidden software error. Lamanna said this error caused Microsoft's servers to become overloaded and stop working when they handled certain types of internet traffic.

该错误与为微软365网络网关服务器进行的软件更新有关。服务器内部的一个进程在多个任务同时尝试使用相同资源时出现问题,这导致服务器停止响应。

The error was linked to a software update made for Microsoft's 365 network gateway servers. A process inside the servers caused problems when several tasks tried to use the same resources at the same time. This made the servers stop responding.

微软的常规测试未能发现该错误。拉曼纳表示,公司在测试中使用的互联网流量类型和服务器配置,与实际客户使用的不同。

Microsoft's regular tests failed to find the error. Lamanna said the company used different types of internet traffic and server setups during testing than those used by actual customers.

该问题也更可能发生在处理能力更强的较新服务器上。然而,拉曼纳说,微软的测试主要集中在处理核心较少的旧款和性能较弱的服务器上。微软365故障:流量增加给服务器带来压力第二个问题是网络流量突然增加。故障发生前大约15小时,微软Teams从备用网络网关切换到了主网关。

The problem was also more likely to happen on newer servers with more processing power. However, Microsoft's tests mainly focused on older and less powerful servers with fewer processing cores, Lamanna said. Microsoft 365 outage: Increased traffic put pressure on serversThe second problem was a sudden increase in network traffic. Around 15 hours before the outage, Microsoft Teams switched from an alternative networking gateway to the primary gateway.

主要网关还支持 Exchange Online,该服务运行 Outlook 的电子邮件服务。此次切换增加了主要网络网关处理的流量,并改变了通过该网关的流量类型。微软表示,此类切换属于其正常运营的一部分,旨在技术故障期间保持服务运行。

The primary gateway also supports Exchange Online, which runs Outlook's email services. This switch increased the amount of traffic handled by the primary networking gateway and changed the type of traffic passing through it. Microsoft said such switches are part of its normal operations and are designed to keep services running during technical problems.

第三个问题是服务器容量减少。由于前一天发生的一个不相关问题,网络网关近 14% 的服务器容量已不可用。微软表示,不可用的服务器容量预计会通过例行维护和健康检查恢复正常。该公司还有足够的额外容量,这意味着减少本身不会导致中断,据 ABC News 报道。

The third problem was reduced server capacity. Nearly 14% of the networking gateway's server capacity was already unavailable because of an unrelated problem that had occurred the previous day. Microsoft said the unavailable server capacity was expected to return to normal through routine maintenance and health checks. The company also had enough extra capacity, meaning the reduction did not cause an outage on its own, according to ABC News.

然而,这三个问题共同造成了一次重大故障。流量增加和服务器容量减少导致网络网关遇到了自 7 月以来一直存在的隐藏软件缺陷。结果,服务器上的请求开始堆积,直到停止响应。拉曼纳表示,这发生在美国互联网流量高峰期。Outlook 中断:超过 80% 的服务器停止响应中断随后从一台服务器蔓延到另一台。微软表示,其流量管理系统通常会在服务器失去连接时将用户转移到其他正常工作的服务器上。据 ABC News 报道,这是一项旨在防止服务中断的标准流程。

However, the three problems together created a major failure. The increased traffic and reduced server capacity caused the networking gateway to encounter the hidden software defect that had been present since July. As a result, requests started building up on the servers until they stopped responding. This happened during a period of peak internet traffic in the United States, Lamanna said. Outlook outage: Over 80% of servers stopped respondingThe outage then spread from one server to another. Microsoft said its traffic management systems normally move users to other working servers when a server loses connectivity. This is a standard process designed to prevent service interruptions, according to ABC News.

然而,在此次事件中,被重定向的流量给剩余的正常工作服务器带来了额外压力。随着更多服务器停止响应,相同数量的流量不得不由更少的服务器处理。这引发了连锁反应,导致更多服务器故障。不到两个小时,隐藏缺陷就影响了微软网络网关系统中的服务器。

However, during this incident, the redirected traffic put additional pressure on the remaining working servers. As more servers stopped responding, the same amount of traffic had to be handled by fewer servers. This created a chain reaction, causing more servers to fail. Within less than two hours, the hidden defect had affected servers across Microsoft's network gateway system.

拉曼纳表示,在故障最严重的时候,超过80%的服务器无响应。微软还表示,其通常的安全系统旨在技术故障期间保持网络不同部分相互隔离,但未能遏制这次中断。这导致故障蔓延至其全球网络网关基础设施。多项微软服务受到这次故障影响。Teams、OneDrive、SharePoint和Office应用出现中断,因为它们连接到了出现故障的网络网关。

At the worst point of the outage, more than 80% of the servers were unresponsive, Lamanna said. Microsoft also said its usual safety systems, which are designed to keep different sections of its network separate during technical problems, failed to contain the disruption. This allowed the outage to spread across its global networking gateway infrastructure. Several Microsoft services were affected by the outage. Teams, OneDrive, SharePoint and Office apps experienced disruptions because they were connected to the faulty networking gateway.

这些服务较早恢复,因为它们能够切换出受影响的网关,改用备用网络网关。微软表示,Teams聊天和会议在大部分故障期间仍可正常使用。然而,并非每项服务都有此选项。支持Outlook电子邮件服务的Exchange Online,无法以同样的方式切换出受影响的网络网关。这使得恢复Outlook更加困难。微软Outlook恢复:为何恢复电子邮件服务耗时更长微软还解释了为何恢复时间比预期更长。

These services were restored earlier because they could switch away from the affected gateway and use the alternative network gateway. Microsoft said Teams chats and meetings continued to work for most of the outage. However, not every service had this option. Exchange Online, which supports Outlook's email services, could not switch away from the affected networking gateway in the same way. This made restoring Outlook more difficult. Microsoft Outlook recovery: Why restoring email services took longerMicrosoft also explained why recovery took longer than expected.

即使工程师让服务器重新上线,这些服务器也立即被故障期间积压的请求淹没。这导致一些服务器再次停止响应。微软的自动系统无法处理如此大规模的服务器故障。这意味着工程师必须采取额外措施才能让服务重新上线。微软还发现,很难撤销该软件更改,因为问题已隐藏数月之久。其用于撤销软件更新和管理新版本发布的常规系统无法快速解决问题。

Even when engineers brought servers back online, they were immediately flooded with requests that had been waiting during the outage. This caused some servers to stop responding again. Microsoft's automatic systems could not handle such a large server failure. This meant engineers had to take extra steps to bring the services back online. Microsoft also found it difficult to undo the software change because the problem had remained hidden for several months. Its normal systems for reversing software updates and managing new releases could not fix the problem quickly.

在发现软件缺陷导致故障后,微软创建了一个新更新,将受影响的软件恢复到早先可用的版本。为安装修复程序,工程师将服务器分组下线,对其进行更新并重新启动,随后再让服务器重新上线。

After finding that the software defect had caused the outage, Microsoft created a new update to bring the affected software back to its earlier working version. To install the fix, engineers took groups of servers offline, updated them and restarted them. They then brought the servers back online.

微软表示,工程师们在其网络的大部分区域谨慎执行了这一过程。有时,他们在一个区域内对超过50%的服务器进行操作。微软已就造成的中断向客户道歉。“我们对给所有客户造成的影响深感抱歉,并认识到这对其正常业务和运营的影响,”拉曼纳在其X平台发帖中写道。微软表示,最终的事后审查(PIR)预计本周晚些时候发布,将提供更多关于此次中断及采取的修复措施的细节。

Microsoft said engineers carried out this process carefully across large parts of its network. At times, they worked on more than 50% of the servers in a region. Microsoft has apologised to customers for the disruption. "We are deeply sorry for the impact to all our customers and recognise the impact that this has on their normal business and operations," Lamanna wrote in his post on X.Microsoft said a final Post-Incident Review (PIR), which will provide more details about the outage and the steps taken to fix it, is expected later this week.

阅读媒体原文 ↗