
几十年来,Top500榜单一直不懈地对全球最大、最强、往往也是最独特的公开已知超级计算机进行排名。随着生成式AI的兴起,该榜单已不再名副其实。中国的“灵犀”超级计算机在高性能LINPACK(HPL)基准测试中实现了2.2 exaFLOPS的双精度计算性能,可能位居这半年一度排名的榜首,但它距离全球最大的计算集群还相差甚远。
For decades, the Top500 list has tirelessly ranked the world’s biggest, most powerful, and often most unique publicly-known supercomputers. With the rise of generative AI, the list no longer lives up to its name. China’s Lineshine supercomputer at 2.2 exaFLOPS of double precision compute in the High-Performance Linpack (HPL) benchmark may be the top system on the bi-annual ranking, but it’s nowhere near the biggest compute cluster in the world.
SpaceXAI的“科洛萨斯”、Meta的“普罗米修斯”以及OpenAI的“星际之门”等超级集群拥有数十万块GPU,规模远超当今Top500榜单上的任何系统。“这些公司对提交Top500榜单不感兴趣,”苏黎世联邦理工学院的Torsten Hoefler本周在超级计算大会发言时表示。这是Top500排名创始人目前正在努力应对的困境。一些人认为,可能是时候重新思考了。
Superclusters like SpaceXAI’s Colossus, Meta’s Promethius, and OpenAI’s Stargate pack hundreds of thousands of GPUs, dwarfing anything on the Top500 today. “These [companies] are not interested in submitting to the Top500,” Eth Zürich’s Torsten Hoefler said during his Supercomputing address this week. It’s a dilemma that the founders of the Top500 ranking are now grappling with. And some believe it may be time for a rethink.
“真正需要的、能更恰当地追踪这一趋势的,不仅是不同的基准测试,还有不同的方法论,”Top500作者之一Erich Strohmaier表示。长期以来,Strohmaier解释说,计算能力大致与计算容量保持一致。过去几年发生在AI领域的情况是,“从市场规模来看,容量真正超越了能力,”他说。“我们过去非常专注于能力,现在依然如此。”
“What’s really needed for tracking that more appropriately is not just a different benchmark but also different methodology,” Top500 author Erich Strohmaier said. For a long time, Strohmaier explains, compute capability was roughly aligned with capacity. What happened in the last few years with AI is “capacity really pulled ahead of capability in terms of market size,” he said. “We were and we still are very focused on capability.”
Strohmaier口中的能力,是指在整个集群上运行单一代码的能力。“我们的基准测试首要规则是,你必须用单一代码对整个系统进行基准测试,不能只是把两部分加起来凑出一个数字,”他解释道。这对于主要大规模云服务商和云提供商如今为助推AI热潮而构建的大型集群来说是个问题。“所有那些系统都是容量型系统,”Strohmaier说。
By capability, Strohmaier is referring to the ability to run one code across the entire cluster. “Our prime rule for benchmarking is, you have to benchmark your whole system with one code and you cannot just add two parts up to make up a number,” he explained. This is problematic for the large clusters major hyperscalers and cloud providers are now building to fuel the AI boom. “All those systems are capacity systems,” Strohmaier said.
它们不是为运行单一代码而设计,而是被架构为运行许多许多较小或中等规模的代码实例,这些实例可能包括训练、微调、强化学习,当然还有推理。
They’re not designed to run one piece of code but have instead been architected to run many, many smaller or medium-sized code instantiations, which may include training, fine-tuning, reinforcement learning, and of course inference.
因此,Strohmaier 表示,指望 SpaceXAI 在 Colossus 这样的集群上运行高性能 Linpack(HPL)是不合理的。但这些集群仍值得追踪。它们只需要一个新的基准测试,既能更好地评判这些阵列的能力,又能防止运营商钻空子。这种基准测试会以何种形态出现,目前尚不得而知。开发新的基准测试或方法绝非易事,修改现有基准测试也非上策。
Because of this, Strohmaier says, it wouldn’t be reasonable to expect SpaceXAI to run something High-Performance Linpack (HPL) on a cluster like Colossus. But these clusters are still worth tracking. They'll just need a new benchmark better suited to judging the capability of these arrays while at the same time preventing operators from gaming the system. What shape or form such a benchmark might take remains to be seen. Developing a new benchmark or methodology is no small task, and changing the existing benchmarks isn’t ideal. ”If we do anything to the benchmark itself, we’re going to lose the historic trends, which is extremely valuable,” Hoefler said.
“如果我们对基准测试本身做任何改动,就会失去历史趋势,而这极其宝贵,”Hoefler 说。允许使用更低精度 —— HPL-MxP 已在一定程度上做到了这一点 —— 是解决这些加速器越来越多地用 FP8 或 FP4 等低精度浮点数据类型换取 FP64 性能这一事实的途径之一。另一条途径是开发单独的 AI 基准测试。这将立即适用于这些集群,但鉴于模型架构变化迅速,构建此类基准测试颇具挑战。
Allowing for lower precisions — HPL-MxP already does this to some extent — is one avenue for addressing the fact that these accelerators are increasingly trading FP64 performance for lower-accuracy floating-point data types like FP8 or FP4. Another avenue is developing a separate AI benchmark. This would be immediately applicable to these clusters, but with the pace of change in model architecture, building such a benchmark is challenging.
“AI 工作负载每两周就会变化,所以如果你今天选定某个东西,顶多一年就会过时,而这些基准测试需要存续数十年,”Hoefler 说。®
“The AI workloads change every two weeks, so if you pick something today, it’s going to be irrelevant in at most a year, and these benchmarks need to live for multiple decades,” Hoefler said. ®