
11月12日周四,微软365服务宕机时,公开解释是分阶段给出的:先是“负载升高”,接着是服务器“意外不可用”,然后是用于“建立安全连接”的流程不堪重负。每一次更新都准确。但没有哪一次更新单独讲清了全貌。
When Microsoft 365 went down on Thursday, Nov. 12, the public explanation arrived in pieces: "elevated load," then servers "unexpectedly unavailable," then an overwhelmed process for "establishing secure connections." Each update was accurate. None of them, on its own, told the story.
微软执行副总裁查尔斯·拉曼纳此后分享了一份摘要,说明他的团队从内部看到的情况。这份摘要描述了大规模计算中最棘手的问题之一:不是不崩溃,而是如何恢复。
Charles Lamanna, an executive vice president at Microsoft, has since shared a summary of what his teams saw from the inside. It describes one of the hardest problems in large-scale computing: not breaking, but coming back.
微软的宕机后来成了一段糟糕一周的开端。11月12日周五,美国联邦航空管理局在全国范围内暂停航班起飞。当晚,服务超过11500家机构的薪酬平台Workday在一次备份测试出错后宕机。11月16日周一,摩根大通的一个软件缺陷拖慢了交易,其他银行的客户也受到了影响。
Microsoft’s outage turned out to be the start of a bad week. On Friday, Nov. 12, the FAA halted departures nationwide. That evening, Workday, the payroll platform for more than 11,500 organizations, went down after a backup test went wrong. On Monday, Nov. 16, a software bug at JPMorgan Chase slowed transactions, and customers at other banks felt it too.
这一周还没有结束。11月17日周二,美国国家安全局的机密内部网络NSA Net出现间歇性中断。消息人士称,没有迹象显示遭到网络攻击,关键通信保持完好,该局还错开了员工到岗时间,以免数万人同时尝试接入。临近中午,苹果证实iCloud登录、App Store和Apple Music出现问题,用户还报告Apple Pay也有故障。数千名Gmail用户报告邮件无法送达,不过谷歌表示没有发现大范围中断的证据。
The week wasn’t over. On Tuesday, Nov. 17, the National Security Agency’s classified internal network, NSA Net, suffered an intermittent outage. Sources said there was no sign of a cyberattack and that critical communications stayed intact, and the agency staggered employee arrival times so tens of thousands of people wouldn’t all try to at once. Around midday, Apple confirmed problems with iCloud sign-in, the App Store and Apple Music, and users reported Apple Pay trouble too. Thousands of Gmail users reported that messages weren’t arriving, though Google said it saw no evidence of a broad disruption.
不同的机构,不同的原因,没有迹象表明它们相互关联。但合在一起,它们说明了一点:一个机构如何恢复,与它为何失败同样重要。微软的恢复过程堪称把这一点做对的典范。
Different organizations, different causes, no sign they were connected. But together they made one point: how an organization recovers matters as much as why it failed. Microsoft’s recovery is a model of getting that part right.
先说没有发生的事。没有邮件丢失,也不是攻击。实际发生的是Exchange Online内部的一次级联故障,这是微软365邮件背后的服务。最严重时,约80%受影响的服务器处于不健康状态。而大多数人忽略的一点是:在服务恢复之前,工程师就已经有了修复方案。最难的工作是重启。
Start with what didn’t happen. No email was lost, and it wasn’t an attack. What did happen was a cascading failure inside Exchange Online, the service behind Microsoft 365 email. At its worst, roughly 80% of the affected servers were unhealthy. And here is the part most people missed: engineers had a fix before service came back. The hardest work was the restart.
级联故障是这样运作的。当一台服务器故障时,它的用户并不会消失。他们的邮件应用会重新连接,负载就落到了邻近的服务器上。如果那些邻近服务器已经接近极限,它们也会跟着倒下。拉曼纳说得直白:一台服务器倒了,它的负载转移到第二台,第二台倒了,然后是第三台。每次重连还需要一次代价高昂的加密“握手”,于是幸存下来的服务器把力气花在让人连进来上,而不是投递邮件。
A cascade works like this. When one server fails, its users don’t disappear. Their mail apps reconnect, and the load lands on the neighbors. If those neighbors were already near their limits, they tip over too. Lamanna described it plainly: one server fell, its load moved to a second, which fell, then a third. Each reconnection also demands a costly encrypted "handshake," so the survivors spent their strength letting people in instead of delivering mail.
压力并非来自更多流量。同样的全球需求只是被挤压到了越来越少的不健康机器上。需求从未改变。改变的是承载需求的机器数量。那么,为什么工程师有了修复方案后,服务没有立刻恢复?
The strain wasn’t caused by more traffic. The same worldwide demand was simply squeezed onto fewer and fewer healthy machines. The demand never changed. The number of machines left to carry it did. So why didn’t service come straight back once engineers had a fix?
电力抢修人员能告诉你答案。长时间停电后,一条线路上的每台暖炉和冰箱都试图同时启动,浪涌可能再次导致线路跳闸。因此,抢修人员会逐段恢复供电。他们称之为冷负荷启动。
Utility crews could tell you. After a long blackout, every furnace and refrigerator on a circuit tries to start at once, and the surge can trip the line again. So, crews restore power section by section. They call it cold load pickup.
微软面对的是数字版本。正如拉曼纳所说,只恢复20%的服务器,它们会立刻再次宕机,因为全世界都在试图连上它们。周四证明了这一点:当天下午恢复的服务器,在晚间流量找到它们后再次故障。
Microsoft faced the digital version. As Lamanna put it, bring back only 20% of the servers, and they go right back down, because the whole world is trying to reach them. Thursday proved it: servers returned that afternoon failed again once evening traffic found them.
周四晚间,一切看似风平浪静。实际上,此次事故中最精细的工作正在进行。工程师逐个区域操作,将每个区域完全离线,再一步到位从零恢复至 100%,以便在全部需求到来前就已备好满负荷容量。日本最先恢复,随后是欧洲、美国及世界其他地区,大致遵循全球各地的工作日顺序。到周五上午,电子邮件队列已基本恢复正常。
Thursday night, it looked as though nothing was happening. In fact, the most delicate work of the incident was under way. Engineers took one region at a time, held it fully offline, then brought it from zero to 100% in a single step, so full capacity was ready before full demand arrived. Japan came first, then Europe, the United States and the rest of the world, roughly following the business day around the globe. By Friday morning, email queues were nearly back to normal.
这种规模的中断极为罕见。如此有纪律的应对更是少见。约三小时内,微软冻结了所有部署和配置变更,确保无人能让事态恶化。它先在少数服务器上测试修复方案再推广,并随着证据变化不断修正公开说明,而非固守最初的判断。
Outages at this scale are rare. Responses this disciplined are rarer. Within about three hours, Microsoft froze all deployments and configuration changes so no one could make things worse. It tested its fix on a few servers before rolling it out, and it kept revising its public explanation as the evidence changed instead of defending a first guess.
最重要的结果从未登上头条。周四无法投递的电子邮件在队列中等待,于周五送达。没有一条消息丢失。这是多年前做出的设计决策在无人预料到需要时发挥的作用。
The most important result never made a headline. Email that couldn’t be delivered Thursday waited in queues and arrived Friday. Not a single message was lost. That is the payoff of design decisions made years before anyone needed them.
教训不在于云端脆弱。而在于行星级规模下,问题不会保持渺小,恢复也不能急于求成。雷德蒙德的工程师们懂得这一点。他们抑制住一次性拨动所有开关的冲动,逐个区域让世界恢复在线。这不是运气。这是工程。
The lesson isn’t that the cloud is fragile. It’s that at planetary scale, problems don’t stay small and recovery can’t be rushed. The engineers in Redmond respected that. They resisted the urge to flip every switch at once and brought the world back one region at a time. That isn’t luck. That’s engineering.