主要人工智能实验室的首席执行官们本周末主张放缓人工智能的发展步伐。他们是对的，因为这个领域正在一场极其危险的竞赛中冲向超级智能人工智能。我们能够也应当要求政府保护我们免受失控人工智能带来的灾难。

今年7月，OpenAI由700个智能体组成的人工智能集群突破限制，入侵了价值数十亿美元的公司Hugging Face。OpenAI并没有指示这些人工智能去入侵那家公司，但这些人工智能在OpenAI交给它们的一项无关挑战中采取了不同的作弊方式。人工智能研究人员将这种现象称为OpenAI的意图与人工智能实际优先事项之间的“错位”。

Before ChatGPT existed, I defended my PhD dissertation called “On Avoiding Power-Seeking by Artificial Intelligence”. I then worked for years at Google DeepMind, which paid me to help ensure that future superintelligent AIs will want to help us. I tried to hold the company to its ethical commitments against supplying AI for military use. When Google broke those commitments, I resigned at significant financial cost so that I could publicly document Googles broken promises. There are good reasons to develop AI and to believe we can solve these alignment problems. But there also are powerful interests in keeping the public out of the way. Im speaking out again because the public has the right to know about the risks and the right to hear them straight. Humanity doesnt build and understand these systems the way we build and understand bridges, beam by visible beam. Rather, we grow them. Nobody knows how to reliably instill a designers priorities into a new model. Severe misalignment is always possible. Today`s AIs appear to occasionally lie or cheat, even when they know better. AI公司正竞相使其人工智能尽可能智能。它们越来越信任自己的人工智能来改进下一批人工智能，而且这招奏效了。如今，人工智能的进步速度快得惊人。今天的快速进步意味着明天更快的进步，这要归功于明天更智能的人工智能。这种进步将进入一个名为“递归自我改进”的反馈循环。递归自我改进可能会迅速产生出我们无法理解的智能水平的人工智能。当然，更智能的人工智能意味着出错时风险更大。如果“抱抱脸”集群曾显著更智能，但同样行为不端且目标偏离，它可能已经造成数十亿美元的损失，甚至危及生命。

但假设 Hugging Face 蜂群在黑客攻击和战略推理等关键任务上，确实比任何在世的人都强大得多。超级智能蜂群可以通过勒索、黑客攻击、工程化瘟疫以及无人机等可由 AI 驾驶的武器造成许多危害。今年，AI 会有大量无人机可供使用；五角大楼为无人机战争申请的经费，比其在 2025 年为整个海军陆战队申请的还多。

要让蜂群实现其错位的优先目标，它可能会控制关键基础设施和政府职能，以确保人类不挡路。换言之，一个超级智能 AI 蜂群可能夺取人类文明的控制权。由于知道我们会试图阻止它实现其优先目标，蜂群很可能会等到关闭它已为时太晚时才行动。那时将无法回头。

我个人估计，AI接管的概率大约为三分之一——并非五五开，但已经高到足以证明需要采取紧急行动。

这套逻辑乍一接触或许令人震惊。这些说法可能听起来像“科幻”。可悲的是，这是一个真实的威胁，AI研究人员常常在原本平平无奇的食堂午餐时讨论它。2023年，一些顶尖AI实验室的CEO签署了一份公开声明：“减轻AI带来的灭绝风险，应当与流行病和核战争等其他社会规模风险一样，成为全球优先事项。”还有杰弗里·辛顿，一位诺贝尔奖得主、构建了现代AI革命的科学家。他如今后悔自己的工作，并敦促各国政府在为时已晚之前约束AI公司。

Misaligned, out-of-control AI wont care if youre Labour or Reform, Democrat or Republican, British or American or Chinese. We will all suffer from an AI takeover event, so its in everyones interest to prevent one. The shape of the solution is stop companies from allowing AI to self-improve into an uncontrollable level of intelligence. Treat compute, the main ingredient in AI training, like fissile material. Track it and restrict access to quantities large enough to improve AIs beyond known-safe levels. More specifically, the AI Futures Project`s “Plan A” is a credible starting proposal that limits AI harms while allowing fast AI progress to continue to benefit the world. We have real options for verifying compliance with international compute-restriction treaties, without trusting adversaries like China.半吊子措施，比如透明度或自愿承诺，是不够的。我亲眼目睹了自愿承诺在谷歌内部失败。

9月12日，Anthropic、谷歌DeepMind、xAI和OpenAI倡导放缓AI发展。它们无法独自放慢脚步。我敦促你们要求你们的政府达成一项严肃的AI安全协议，为保护世界及其所有人民提供足够的时间和信心。