Meta（扎克伯格公司）本周披露，位于俄亥俄州新奥尔巴尼的 Prometheus 超级集群中，超过 11% 的 GPU 处于“损坏或闲置待命”状态，以便在下一块 GPU 掉线时顶上。这个庞大的冗余 GPU 池只是这家超大规模云服务商为维持 AI 训练运行而采取的多项措施之一。
More than 11 percent of the GPUs at Meta’s Prometheus supercluster in New Albany, Ohio, are either dead or sitting idle waiting to step in when the next drops out, Zuckercorp revealed this week. The massive pool of redundant GPUs is just one of several measures the hyperscaler has taken to keep its AI training runs on track.
Prometheus 是 Meta 迄今最大的算力部署之一。该超级集群采用 Nvidia Grace Blackwell 机架系统（以及即将推出的 Vera Rubin 机架系统）供电，Meta 表示最终将扩展至 1 吉瓦容量，涵盖约 100 万张加速器。但正如所有在大规模算力集群上工作过的人所深知——无论是用于大语言模型训练还是传统 HPC（高性能计算）工作负载——芯片越多，问题必然越多。
Prometheus is among Meta's largest compute deployments to date. Powered by Nvidia's Grace Blackwell and (soon) Vera Rubin rack systems, Meta says the supercluster will eventually scale to a gigawatt of capacity spanning about a million accelerators. But as anyone who has worked on a large scale compute cluster understands, whether for LLM training or traditional HPC workloads, more chips inevitably means more problems.
在单个 8-GPU 节点中，GPU 故障概率相对较低；但在 10 万张 GPU 的集群中，故障不可避免。Meta 正在构建的规模是其 10 倍，且需要跨多个数据中心进行分布式训练。让整个系统持续稳定运行、足以到达下一个检查点，成为一大棘手难题。
In an eight-GPU node, the chance of having a GPU fail is relatively low. In a 100,000-GPU cluster, failures are unavoidable. Meta is building something that's 10x larger and requires distributed training runs across multiple datacenters. Keeping the entire thing humming long enough to reach the next checkpoint becomes a major headache.
Meta 在本周发布的一篇博客文章中，独家展示了其为防止故障拖垮 Muse Spark 等模型及其最新发布的 Muse Wonder 的开发所做的工作。
In a blog post published this week, Meta offered a unique look at the work it's done to keep failures from derailing the development of models like Muse Spark or its most recent release Muse Wonder.
闲置 GPU 是持续运行的代价 Meta 表示，在新奥尔巴尼 Prometheus 部署的第一阶段，公司每 8000 张 GPU 每天大约遭遇 1000 次中断。中断不等于 GPU 故障，仅指某些组件崩溃、失效或超时，迫使训练运行停止并回滚至最近一次已知的良好检查点。Meta 称，一年来已将中断次数降低 99%，降至每 8000 张 GPU 每天约 11 次。实现这一目标的途径之一，就是保留部分 Prometheus GPU 作为储备。
Idle GPUs are the cost of continuity When Meta began the first phase of its Prometheus deployment in New Albany, the company says, it was experiencing roughly 1,000 interruptions per day for every 8,000 GPUs. Interruptions don't mean GPU failure. They simply mean something crashed, failed, or timed out, forcing the training run to stop and roll back to the last known good checkpoint. Meta claims in the year since it's been able to cut down the number of interruptions by 99 percent, to about 11 per day per 8,000 GPUs. One of the ways it's done this is by keeping some of Prometheus' GPUs in reserve.
该集群目前由 GB200 和 GB300 系统组合供电，每个 NVLink 域包含 72 块 GPU。然而，任意时刻仅有 64 块在实际使用。其余 8 块本质上是热备件。Meta 这样做是因为单块 GPU 故障会导致整个机柜瘫痪，从而中断训练任务。现在，当 GPU 故障时，备件之一可接管其位置。虽然这节省了时间 —— Meta 无需将整个机柜从集群中剔除，也不必等待人员更换故障部件才能重新开始训练 —— 但 GPU 故障仍会导致中断，只是中断时间更短。
The cluster is currently powered by a combination of GB200 and GB300 systems, each with 72 GPUs per NVLink domain. However, only 64 of them are ever in use at any given moment. The other eight are essentially hot spares. Meta did this because a single failed GPU would take out an entire rack and interrupt the training run. Now, when a GPU fails, one of the spares can slot in and take its place. While this saves time — Meta doesn't have to drop the entire rack from the cluster or wait for someone to swap the faulty component to start training again — a failed GPU would still result in an interruption, albeit a shorter one.
并行弹性为缓解这一问题，Meta 利用了大规模训练作业通常采用数据并行这一特点，即集群内驻留着多个模型副本。“当故障可被隔离到某个数据并行模型副本时，我们允许单个故障副本下线，而集群其余部分重复该步骤并继续高效训练，从而降低硬件故障的影响，”Meta 解释道。“受损分片只需在可用的备用主机上重启并重新同步，即可将潜在的整体系统停机转化为聚合吞吐量的轻微、暂时性下降。”
Parallel resiliency To mitigate this, Meta took advantage of the fact that large scale training jobs usually use data parallelism, where multiple copies of the model are resident within the cluster. "When a fault can be isolated to a data-parallel model replica, we reduce the impact of a hardware failure by allowing a single faulty replica to be taken offline, while the remaining cluster repeats that step and continues productive training," Meta explained. "The compromised slice simply restarts on available spare hosts and resynchronizes, converting a potential total system halt into a minor, temporary reduction in aggregate throughput."
换言之，只有当故障超出数据并行域时，Meta 的训练运行才会中断。遇到这种情况，公司已实施一套全面的检查点机制，可将其类比为自动保存或版本历史。这样一旦发生故障，Meta 就不必从头开始。
In other words, Meta's training runs only get interrupted if the fault exceeds the data parallel domain. When this happens, the company has implemented a comprehensive checkpointing scheme, which you can think of a bit like auto-save or a revision history. That way if something does fail, Meta doesn't have to start over from scratch.
“我们的分布式检查点架构将模型状态 —— 包括梯度和优化器数据 —— 异步保存到多层存储中，”公司写道。NVLink 成长的烦恼Meta 表示，最困难的挑战之一是让用于连接 72 块 GPU 的 NVLink 结构表现得像一个巨大的单一加速器。
"Our distributed checkpointing architecture asynchronously saves the model state — including gradients and optimizer data — to multi-tier storage," the company wrote. NVLink growing pains Meta says one of the most difficult challenges was making the NVLink fabrics used to connect 72 GPUs behave like one great big accelerator.
早期基于 H100 和 H200 的系统每个 NVLink 域仅配备八张 GPU，这意味着互连链路更短、数量更少。但随着 Nvidia Blackwell 架构的推出，这一功能扩展了九倍。公司写道：“拥有超过 60 万条 NVLink 连接，即使是边缘性的信号衰减也会触发频繁的 NVLink 恢复周期。”为解决这一问题，Meta 不得不实施持续遥测系统，并在系统投入生产前进行 L11 诊断烧机，以隔离薄弱链路。
Earlier H100 and H200-based systems featured just eight GPUs per NVLink domain, which meant the links were far shorter and less numerous. But this functionality was expanded nine-fold with Nvidia's Blackwell generation. "With over 600,000 NVLink connections, even marginal signal degradation triggers frequent NVLink recovery cycles," the company wrote. To overcome this, Meta had to implement a continuous telemetry system and conduct L11 diagnostic burns to isolate weak links before the systems were put into production.
电力是另一大挑战。为训练运行而开启和关闭数万张 GPU 会产生被称为瞬态的巨大功耗峰值，可能使电源供应设备不堪重负。据 Meta 称，通过专注于集群的稳定性和可靠性，其在最新模型 Muse Wonder 上实现了预训练超过 85% 的“良吞吐率”——即有效吞吐量的衡量指标——以及强化学习超过 90% 的良吞吐率。公司的雄心不止于此。
Power was another challenge. Ramping up and down tens of thousands of GPUs for a training run can introduce massive spikes in power consumption called transients, which can overwhelm power supply equipment. According to Meta, by focusing on the stability and reliability of its cluster, it was able to achieve more than 85 percent "goodput" — a measure of useful throughput — for pretraining and more than 90 percent for reinforcement learning on its latest model Muse Wonder. The company's ambitions don't stop here.
在朝着百万张 GPU 的目标迈进的过程中，Meta 希望将中断率降至每 8000 张 GPU 每天仅两次，即平均故障间隔约 10 万 GPU 小时。®
As it works its way toward a million GPUs, Meta hopes to get its interruption rate down to just two per day for every 8,000 GPUs, or about 100,000 GPU-hours mean time between interrupts. ®