Meta位于俄亥俄州新奥尔巴尼的Prometheus超级集群已在多栋建筑内运行数十万块Nvidia GPU，Meta计划将其扩展至约100万块。该公司周一在一篇工程博客文章中披露了细节。
Meta’s Prometheus supercluster in New Albany, Ohio, already runs hundreds of thousands of Nvidia GPUs across several buildings, and Meta plans to grow it to about 1 million. The company set out the details in an engineering blog post on Monday.
这个吉瓦级集群的第一阶段已建成完工。Meta曾用它训练其Muse系列模型，包括Muse Wonder模型。两年前，该场地还是一个运行网络流量的传统数据中心。
The first phase of the gigawatt-scale cluster is finished. Meta used it to train its Muse family of models, including the Muse Wonder models. Two years ago, the site was a conventional data centre running web traffic.
Meta写道：“在全规模下，Prometheus将由大约100万块分布式GPU组成，所有GPU协同运作。”
“At full scale, Prometheus will comprise roughly 1 million distributed GPUs, all functioning in lockstep,” Meta wrote.
三代Nvidia机架 Prometheus目前运行两个Nvidia平台，两者均围绕每机架72块GPU构建，作为单一加速器运行。Catalina使用GB200 NVL72，Clemente使用更新的GB300 NVL72。第三个平台San Miguel将引入Nvidia的Vera Rubin NVL72机架。
Three generations of Nvidia racks Prometheus runs two Nvidia platforms today, both built around racks of 72 GPUs that act as one accelerator. Catalina uses the GB200 NVL72, and Clemente uses the newer GB300 NVL72. A third platform, San Miguel, will add Nvidia’s Vera Rubin NVL72 racks.
GPU部署在传统数据中心、AI优化建筑和快速部署结构中，由数英里光纤互联。每栋建筑的网络最多连接20,736块GPU。一个独立的汇聚网络将各建筑连接起来，跨度从2公里到100多公里不等。Meta还使用从合作伙伴租赁的产能。
The GPUs sit in traditional data centres, AI-optimised buildings and rapid-deployment structures, linked by miles of fibre. Each building’s network connects up to 20,736 GPUs. A separate aggregation network joins the buildings over distances from 2km to more than 100km. Meta also uses capacity leased from partners.
从每天1,000次中断 在如此规模下，单台机器的故障即可导致整个训练运行停止。Meta表示，2025年中期第一阶段期间，中断峰值曾达到每8,000块GPU每天近1,000次事件。一年的工程工作已将这一数字降低99%。
From 1,000 interruptions a day At this scale, a fault on a single machine can stop a whole training run. In mid-2025, during the first phase, interruptions peaked at nearly 1,000 events per 8,000 GPUs each day, Meta said. A year of engineering work has cut that by 99%.
早期最大的问题是GPU之间60多万个NVLink连接的信号质量。随着Meta扩大GB300机架规模，内存故障成为更大的问题。
The biggest early problem was signal quality on more than 600,000 NVLink connections between GPUs. Memory faults became a larger issue as Meta scaled up the GB300 racks.
在Muse Wonder训练期间，集群将其预训练时间的85%以上保持为高产时间，强化学习阶段则超过90%。平均每次中断间隔约47,000 GPU小时，固件修复后已提升至54,000小时以上。Meta的目标是达到约100,000小时。
During Muse Wonder’s training, the cluster kept more than 85% of its pre-training time productive, and more than 90% for reinforcement learning. It ran about 47,000 GPU hours on average between interruptions, now above 54,000 after firmware fixes. Meta’s goal is about 100,000.
可靠性也需要硬件成本。每座配备72个图形处理器（GPU）的机架运行64个GPU的区块，并保留8个GPU作为备件，Meta还保留了大约2%的机架作为储备。因此，在32,768个GPU上进行的一次训练运行需要37,656个GPU，大约多出13%。
Reliability also costs hardware. Each 72-GPU rack runs a 64-GPU slice and keeps eight GPUs as spares, and Meta holds about 2% of racks in reserve. A training run on 32,768 GPUs therefore needs 37,656, about 13% more.
高达数兆瓦的功率波动 当数万个GPU同步计算和通信时，丛集的功耗会波动30%到40%。启动或暂停一项大型作业可以在几秒钟内使功率发生数兆瓦的变化。Meta利用其称为Dynamo Dimmer的软件逐渐限制GPU的功率，并有基于机架、设施和电网层面的电池作为后盾。
Power swings of tens of megawatts When tens of thousands of GPUs compute and communicate in step, the cluster’s power draw swings by 30% to 40%. Starting or pausing a large job can shift tens of megawatts in seconds. Meta caps GPU power gradually with software it calls Dynamo Dimmer, backed by batteries at rack, facility and grid level.
Meta表示，它支付了其在新奥尔巴尼使用的能源的全部成本，并以新的清洁和可再生能源进行匹配。7月，随着其针对人工智能的天然气基建发展速度超过可再生能源，该公司退出了RE100倡议。
Meta says it pays the full cost of the energy it uses in New Albany and matches it with new clean and renewable energy. In July, it left the RE100 pact as its gas build-out for AI outpaced renewables.
Meta表示，其下一个超级丛集——位于路易斯安那州里奇兰县的Hyperion，在设计时借鉴了普罗米修斯的经验教训。Meta还已向数据中心社区承诺出资10亿美元。
Its next supercluster, Hyperion in Richland Parish, Louisiana, is being designed with lessons from Prometheus, Meta said. Meta has also pledged $1bn to data centre communities.