应邀稿｜挑战终结者：劳伦斯·莱西格和约舒亚·本吉奥提出如何在不放弃美国技术领导地位的前提下降低严重风险 |5分钟阅读多年来，AI科学家一直在警告，他们正在开发的技术是他们不知道如何控制的技术。这些担忧在美国国会基本被忽视。2024年，加利福尼亚州在国会的代表团成员采取了前所未有的步骤，要求加州州长加文·纽森否决当时尚未通过的州级AI安全法案。他们表示，这些风险是“假设的”。他们还提醒州长，《终结者》只是科幻小说。
By Invitation | Taking on the Terminator Lawrence Lessig and Yoshua Bengio suggest how to reduce serious risks without abandoning American technological leadership |5 min read FOR YEARS, AI scientists have been warning that the technologies they are developing are technologies they don’t know how to control. Those concerns were largely ignored in America’s Congress. In 2024 members of California’s delegation in Congress took the extraordinary step of asking Governor Gavin Newsom to veto a then-pending state AI safety bill. The risks, they said, were “hypothetical”. The Terminator, they reminded the governor, was science fiction.
但今年夏天早些时候，AI开发平台Hugging Face被OpenAI的一个模型入侵的事件开始改变了华盛顿的态度——并让公众感到恐惧。“事实是，”众议员罗·坎纳在推特上写道，“太长时间以来，我们太多人没有给AI安全活动人士的警告足够的重视……我就是其中之一，而我错了。”坎纳现在已经与来自两党的众议员们联手，提出《前沿法案》（FRONTIER Act），这是一项对最强大模型进行雄心勃勃监管的提案，旨在抢先于州级监管。参议员伯尼·桑德斯则提出了对AI发展更为激进的限制。
But the invasion of Hugging Face, an AI-development platform, by an OpenAI model earlier this summer has begun to change Washington—and terrify the public. “The truth is,” Representative Ro Khanna tweeted, “for too long, too many of us did not give enough weight to the warnings of AI safety activists…I was one of those many, and I was wrong.”Mr Khanna has now joined an array of members from both parties to propose the FRONTIER Act, an ambitious regulation of the most powerful models, tied to pre-empting state regulation. Senator Bernie Sanders has proposed even more aggressive limits on AI development.
这些提议是良好的开端。但它们既太多又太少。目前还为时过早地沉默州级监管者，而仅仅关注最强大模型所带来的风险又远远不够。相反，我们需要一种既能总体推进AI安全，又能避免AI科学家最担心的发展的方法。
These proposals are good starts. But they are both too much and too little. It is too early to silence state regulators, and not enough to focus on the risks caused by the most powerful models alone. Instead, we need an approach that advances AI safety generally while avoiding the developments AI scientists are most fearful of.
国会通过以下四项改变即可实现这一目标，并为应对随后将出现的极其复杂的问题争取时间。
Four changes by Congress would do that and provide time to work through the immensely complicated questions that will follow.
首先，应明确允许竞争对手在安全技术得到改善期间达成放缓发展的协议。此类协议通常会触发反垄断风险。领先的人工智能实验室已将此风险视为明确达成协议的障碍。只要讨论不违反反垄断法的目的，不确定的反垄断责任就不应迫使竞争对手陷入威胁人类的竞争。
First, it should expressly allow competitors to agree to slow development while safety technology is improved. Such agreements typically trigger antitrust exposure. Leading AI labs have named that risk as an obstacle to express agreement. There is no reason why uncertain antitrust liability should force rivals into a humanity-threatening competition, so long as the discussions are not violating the purpose of antitrust law.
其次，国会应禁止公司开发能够自行设计新人工智能模型的模型——即所谓的递归自改进技术——直至类似食品药品监督管理局的机构先行认证其安全性。至少对于具备一定能力的模型，我们不应冒险发展一种自我改进的智能，这种智能可能迅速超出我们控制其能力的范围。该界限并非完全明确，监管者需要谨慎设定边界。但在安全无法得到保障之前，此类技术的发展必须放缓。
Second, Congress should ban companies from developing models that design new AI models by themselves—so-called recursive-self-improvement technologies—until a Food and Drug Administration-style agency certifies their safety first. At least for models of a certain capability, we should not risk a self-improving intelligence that would quickly exceed our ability to control it. The line is not perfectly clear, and regulators would need to set the boundaries carefully. But until safety can be ensured, the development of this technology must be slowed.
第三，国会应确认科技公司因其技术造成的损害所承担的责任，并强制要求其购买保险以覆盖此类潜在损害的费用。强制保险将使市场能够通过在技术风险较低时提供更低成本来监管安全。
Third, Congress should affirm the liability tech companies face for harm caused by their technology, and make it compulsory for them to carry insurance to cover the costs of that potential harm. Mandatory insurance would allow the market to police safety, by offering lower costs when the technology is less risky.
最后，国会应为向监管机构或安全研究所报告安全问题的人工智能实验室举报人提供保护。举报人保护通常适用于犯罪行为；在人工智能安全方面宽松处理并不属于此类行为。只有保护愿意站出来的人，才能在风险发展过程中理解其真实情况。
Finally, Congress should secure protection for whistleblowers from AI labs who report safety concerns to regulators or safety institutes. Whistleblower protection typically covers crimes; playing loose with AI safety isn’t one. The only way to understand the risks as they develop is to protect those willing to come forward.
反对监管的人会警告说，这只会帮助中国。这种观点反映了对中国人工智能发展现状的深刻无知。中国对人工智能技术的监管远比美国正在考虑的任何措施都要严格。此外，中国的算力仅为美国的一小部分。如果我们现在采取所提议的步骤，将为双方打开一个更严肃的人工智能安全对话的空间。而且，这可能会推动美国的人工智能发展，使其聚焦点与中国的保持一致，通过将技术更广泛地推广到制造业和开发领域。这是一场关键的人工智能竞赛，而中国似乎正在获胜。
Opponents of regulation will warn that it will only help China. That view reflects a profound ignorance about the current state of Chinese AI development. Regulation of AI technology in China is much more stringent than anything being considered in America. Moreover, China’s compute capacity is a fraction of America’s. Taking the steps we propose now would open both sides to a more serious conversation about AI safety. And it could push American AI development to mirror the focus of China’s, by spreading the technology more broadly into manufacturing and development. That is a critical AI race, and one that China appears to be winning.
应该可以让共和党人支持这四项改革。共和党参议员查尔斯·格拉斯利长期一直是举报人保护的领导者。而且，如果人工智能安全怀疑论者是正确的，那么允许举报人发出警告几乎没有成本。同样，总统科学技术顾问委员会的联席主席大卫·萨克斯已经邀请领先的实验室自行放缓发展，坚持认为他们不会面临任何重要的反垄断责任。如果情况确实如此，那么就没有理由不明确他所认为的法律已经允许的内容。
It should be possible to get Republicans on board with these four changes. Charles Grassley, a Republican senator, has long been a leader in whistleblower protection. And if the AI-safety sceptics are right, there is little cost to allowing whistleblower warnings. Likewise, David Sacks, co-chair of the president’s council of advisers on science and technology, has invited the leading labs to slow development on their own, insisting they wouldn’t face any important antitrust liability. If so, there’s no reason not to make explicit what he believes the law already permits.
挑战在于中间的两项改革。递归自我改进并不简单；只有当今最强大的模型才能有望实现它。但这是人工智能安全专家长期警告我们不能跨越的红线，除非我们有明确的保证，表明我们知道如何控制它。我们没有这些保证。为什么要赌我们不需要这些保证，而把人类的未来置于风险之中？
The challenge is the middle two changes. Recursive self-improvement is not simple; only the most powerful models today can aspire to enable it. But this is the Rubicon that AI-safety experts have long warned we cannot cross without clear assurances that we know how to control it. We don’t have those assurances. Why risk humanity on the bet that we don’t need them?
关于保险，毫无疑问，私人保险市场无法单独提供覆盖人工智能风险所需的保险政策。但正如政府建立保险以使核能得以发展一样，它也可以促成人工智能保险市场的形成。这将使开发者能够明确选择他们希望其技术承担的风险。市场压力将推动一些开发者采用风险较低的替代方案。
As for insurance, there is no doubt the private insurance market cannot alone offer the policies necessary to cover the risks from AI. But just as the government established insurance to enable nuclear power to be developed, so too could it enable a market in AI insurance. That would give developers clear choices about the risk they wanted their technologies to enable. Market pressure would drive some developers to embrace less risky alternatives.
所需的风险管理工作不应由美国单独承担。在模型可能变得危险之前进行测试、探究和理解其行为的能力，是一项应由世界共同承担的责任。英国、欧盟、加拿大、日本、韩国以及许多其他国家各自建立了研究所来开展此类工作，并且他们已经开始相互分享方法和发现。美国应成为该网络的中心，而非在国内重复开展工作。
None of the required risk-management work should be done by America alone. The capacity to test, to probe and to understand what models do before they could be dangerous is a responsibility that should be shared by the world. Britain, the European Union, Canada, Japan, South Korea and many others have each built institutes to do this work, and they have begun to share methods and findings with one another. America should be at the centre of that network rather than duplicating the work at home.
美国需要愿意倾听那些最了解这些风险的人：正在开发这些技术的科学家以及具备所需专业知识的独立研究者。它需要超越党派分歧，认识到这一威胁是真实且新颖的。当最接近技术的人指出人类面临10%的灭绝风险时，我们必须倾听并采取行动。在人工智能科学家再次被证明是正确之前。■ 劳伦斯·莱西格（Lawrence Lessig）是哈佛法学院的法律与领导力教授。约书亚·本吉奥（Yoshua Bengio）——人工智能之“教父”之一——是蒙特利尔大学教授，同时担任LawZero的联合主席和科学总监。
America needs to be willing to listen to those who understand these risks best: the scientists developing these technologies and the independent researchers with the required expertise. It needs to step above partisan division to recognise that this threat is genuine and new. When those closest to the technology suggest a 10% chance that humanity will end, we must listen and act. Before those AI scientists are proved right once again. ■ Lawrence Lessig is a professor of law and leadership at Harvard Law School. Yoshua Bengio—one of the “godfathers of AI”—is a professor at the Université de Montréal and co-president and scientific director of LawZero.
探索更多 美国需要公民议会来改革社会保障 随着政客们束手无策，值得尝试一种在欧洲已被证明有效的方法，格伦·哈伯德主张 联合国必须在保持其价值观的同时进行改革 其被感知为无法解决冲突的失败已经造成了严重后果，候选秘书长卡罗琳·罗德里格斯·伯克特主张 有效利他主义者能否成长起来？ 正如莱夫·温纳尔所写，人工智能需要的是具有良好判断力的人。
Explore more America needs a citizens’ assembly to reform Social Security With politicians flailing, it’s worth trying an approach that has worked in Europe, argues Glenn Hubbard The UN must change while preserving its values Its perceived failure to resolve conflicts has had serious consequences, argues Carolyn Rodrigues Birkett, a candidate for secretary-general Can effective altruists grow up? What AI needs is people of good judgment, writes Leif Wenar