字号·· | 护眼
纽约时报

微软全球宕机后紧急恢复上线的内幕Inside Microsoft’s Rush to Get Back Online After a Global Outage

点「原文对照」整页切到原文,或双击某段只看那段的原文。

微软承认,约有20项不同的服务因这次故障受到影响。图片来源……格兰特·欣兹利为《纽约时报》拍摄Microsoft acknowledged that about 20 different services were impaired by the outage. Credit... Grant Hindsley for The New York Times

一场持续时间较长的故障导致许多人无法使用电子邮件和其他服务,这再次提醒人们,世界的技术基石依赖于极少数公司。

A prolonged outage left many without email and other services, the most recent reminder that the world’s technological underpinnings rely on just a few companies.

周四深夜,数百名微软工程师和高管坐在漆黑的办公室或家中的地下室里,登录视频会议,应对一家科技公司的噩梦场景。

Late into the night Thursday, hundreds of Microsoft engineers and top executives were sitting in their darkened offices or home basements, logged into a video conference triaging a nightmare scenario for a technology company.

世界仍在依靠电子邮件运转,而其中大部分是通过微软Outlook处理的。数小时以来,公司故障影响了医院、学校、企业以及华盛顿的国会办公室。甚至微软自己的员工也无法使用。在视频会议上,负责监管公司计算基础设施的最高级别高管Scott Guthrie提出了可能的修复方案,一位知情人士透露,该人士未获授权公开发言。人们咬着指甲、揉着额头,争论如何关闭其他服务以防止“蜂拥而至”,以便当系统恢复上线时,不会因新需求涌入而不堪重负并再次关闭。

The world still runs on email, and much of that is processed through Microsoft Outlook. For hours, a company outage affected hospitals, schools, businesses and Congressional offices in Washington. Even Microsoft’s own staff couldn’t use On the videoconference, Scott Guthrie, the senior most executive overseeing the company’s computing infrastructure, was proposing possible fixes, according to a person familiar with the call who was not authorized to speak publicly. People chewed on fingernails, rubbed their foreheads and debated how to turn off other services to prevent a “stampeding herd” so that as systems come back online, they would not be overwhelmed with new demands and shut back down.

到周五上午,公司已至少道歉两次。公司表示,确定问题源于最近的一次网络升级,该升级导致部分服务器关闭,且系统正在稳步改善。公司在一份声明中说:“鉴于需求规模,我们逐步恢复了服务,在每个阶段都验证了稳定性。虽然这花费了更长时间,但有助于确保稳定恢复。”最终,他们挡住了“蜂拥而至”的局面。

By Friday morning, the company had apologized at least twice. It said it determined that the problem stemmed from a recent upgrade to its network, which shut down some servers, and that the systems were steadily improving. “Given the scale of demand, we restored services gradually, validating stability at each stage,” the company said in a statement. “While this took longer, it helped ensure a stable restoration.”Finally, they had kept the herd at bay.

这场持续时间长、范围广的故障,只是最近一次提醒,现代生活和商业的技术基石由极少数大公司运行。虽然网络攻击和恶意行为者可能针对其系统,但有时这些公司不得不通过惨痛教训认识到,在更新或维护自己的全球系统时,它们可能会引入毁灭性的缺陷。

The prolonged and widespread outage was just the most recent reminder that the technological underpinnings of modern life and business are run through just a handful of major companies. And while cyberattacks and malicious actors can target their systems, sometimes the companies learn the hard way that they can introduce devastating flaws when updating or maintaining their own global systems.

医院、执法部门、政府机构及其他关键服务往往处于脆弱状态——无法运作数小时甚至数天。

Hospitals, law enforcement and government agencies and other vital services are often left vulnerable — unable to function for hours or days at a time.

去年,亚马逊云计算服务中断了15小时,导致包括Zoom和《纽约时报》部分业务在内的一大批应用程序和网站瘫痪。而在2024年夏季,网络安全公司CrowdStrike的一次错误软件更新导致航空公司停飞航班、企业提前关门、手术被迫重新安排。

Last year, Amazon’s cloud computing services were down for 15 hours and knocked a slew of apps and websites, including Zoom and parts of The New York Times, offline. And in the summer of 2024, a faulty software update by the cybersecurity firm CrowdStrike caused airlines to ground flights, businesses to close early, and surgeries to be rescheduled.

对微软而言,本周故障带来的麻烦尚未结束。投资银行威廉·布莱尔的分析师周五在研报中写道,短期财务影响应当是“可控的”。但他们表示,这次持续的故障是一次声誉打击,可能会让企业和其他客户重新考虑是否要将如此多的业务押注在单一技术供应商身上。

For Microsoft, the problems from this week’s outage are not done yet. The short-term financial fallout should be “manageable,” analysts at the investment bank William Blair wrote in a research note Friday. But they said this prolonged outage is a reputational hit that could make companies and other customers reconsider having so much riding on one technology provider.

“我们认为,此次故障可能会促使企业重新评估其对微软基础设施的依赖程度,”他们写道。

“We believe the outage could prompt enterprises to reassess their dependence on Microsoft’s infrastructure,” they wrote.

受影响的Outlook及其他服务,包括Teams及其AI助手Copilot,均属于Microsoft 365云订阅业务的一部分。该业务拥有超过4.5亿用户,在截至9月30日的上一季度收入近280亿美元。网络迷因戏称该产品应改名为“Microsoft 364”。

Outlook and other services affected, including Teams and its A.I. assistant Copilot, are part of Microsoft 365’s cloud subscription business, which is used by more than 450 million people and brought in almost $28 billion in its last quarter, which ended Sept. 30. Online memes joked the offering should be rebranded as Microsoft 364.

这些订阅业务是公司业务的核心,也是其推广AI工具的基石。微软向客户主张,如果他们在微软系统上存储更多数据,就能更轻松地实施AI并最大化其效益。微软承认,约20项不同服务受到故障影响,这提醒人们:整合供应商也可能制造出更大的单点故障。

These subscriptions are central to the company’s business and the foundation on which it is promoting A.I. tools. Microsoft makes the case to customers that if they keep more data on Microsoft’s systems, they will be able to implement and maximize the benefits of A.I. more easily. Microsoft acknowledged that about 20 different services were impaired by the outage, a reminder that consolidating vendors can also create a bigger single point of failure.

微软于周四上午首次发现Outlook问题,当时美国西海岸时间约为上午9点,微软总部即位于该时区。故障随后从美洲大陆蔓延至全球。

Microsoft first detected the Outlook problems Thursday morning, around 9 a.m. on the West Coast, where the company is headquartered. It soon spread from the Americas around the world.

起初,该公司表示认为原因是流量意外激增。随后,它确定公司数据中心内大量服务器宕机,导致其余系统承受压力。即使部分服务器恢复上线,它们又会再次关闭,就像打地鼠游戏一样。

At first, the company said it thought the cause was an unexpected spike in traffic. Then it determined a large number of the company’s servers in its data centers were down, straining the rest of the system. Even as some servers were brought back online, they would shut down again, like a game of Whac-A-Mole.

英国伯明翰大学网络安全助理教授卢卡·阿纳博尔迪表示,此次中断提醒人们,当人们使用“云”时,实际上依赖的是服务器,而服务器的资源是有限的。

Luca Arnaboldi, an assistant professor of cybersecurity at the University of Birmingham in England, said the outage was a reminder that when people use “the cloud,” they are actually relying on servers, which have finite resources.

在当晚的视频会议上,古斯里表示,虽然其他系统也宕机了,但团队应优先恢复 Outlook。负责微软许多生产力服务的查尔斯·拉马纳也在场。他已公开道歉,并表示微软已排除网络攻击是罪魁祸首。罪魁祸首是公司自身的系统。

On the overnight videoconference, Mr. Guthrie said while other systems were down too, the teams should prioritize restoring Outlook. Charles Lamanna, who heads many of Microsoft’s productivity services, was there too. He had already publicly apologized and said Microsoft had ruled out a cyberattack as the culprit. The company’s own systems were to blame.

伊莎贝拉·夸伊在伦敦采写了本报道。迈克尔·莱文森在纽约采写了本报道。

Isabella Kwai contributed reporting from London. Michael Levenson contributed reporting from New York.

凯伦·韦斯为《纽约时报》撰写科技报道,常驻西雅图。她的报道聚焦于亚马逊和微软,这两家美国最强大的公司。

Karen Weise writes about technology for The Times and is based in Seattle. Her coverage focuses on Amazon and Microsoft, two of the most powerful companies in America.

阅读媒体原文 ↗