
保罗·克里斯蒂亚诺是OpenAI基金会董事会成员,也是该基金会安全委员会成员。他表示,AI开发者现在应当停止改进训练算法,因为他们的安全措施太薄弱,无法阻止窃取。他在11月16日播出的一档播客节目中向德瓦克什·帕特尔阐述了这一观点。
Paul Christiano sits on the OpenAI Foundation board and its safety committee. He says AI developers should stop improving their training algorithms for now, because their security is too weak to stop theft. He made the case to Dwarkesh Patel on a podcast published on 16 November.
“我要说,未来一年我们的风险看起来更像是10%,而不是1%,”克里斯蒂亚诺在谈到AI接管时说。
“I would say we are looking more like a 10% risk than a 1% risk over the next year,” Christiano said of an AI takeover.
他说,如果开发者不加防范地一路狂奔,风险“明显超过”10%。他还认为,除非整个行业放慢脚步,否则一年内出现超级智能的几率“也许是三分之一”。
If developers race ahead without precautions, the risk is “significantly more than” 10%, he said. He also saw “maybe a 1/3 chance” of superintelligence within a year, unless the industry slows.
他强调,他是以私人公民身份发言的。他说,他并不是在转达董事会的观点。
He stressed that he spoke as a private citizen. He was not reporting the board’s views, he said.
主张暂停的理由:克里斯蒂亚诺说,各实验室不应在现有安全水平下继续改进预训练和强化学习。在他看来,这对美国国家安全不利。如果实验室日后为了安全而放慢速度,对手就可能窃取模型或方法,从而抢先一步。
The case for a pause Christiano said labs should not keep improving pre-training and reinforcement learning with their current security. That is bad for US national security, in his view. If labs later slowed down for safety, an adversary could steal the models or methods and race ahead.
他说,OpenAI和Anthropic最近内部系统中都出现了“极其不高明的行为者”。这些人接触到了Slack或OpenAI的共享代码,其权限足以提取大部分算法工作成果。帕特尔提到,有研究人员利用Claude接触到了OpenAI的内部代码。
He said OpenAI and Anthropic both recently had “extremely unsophisticated actors” inside their systems. Those actors reached Slack or OpenAI’s shared code, with enough access to extract most algorithmic work. Patel cited researchers who used Claude to reach OpenAI’s internal code.
“改进算法这件事本身就很容易扩散,即使在正常情况下也是如此,而且可能被迅速窃取。我认为他们现在就应该停止这样做,”克里斯蒂亚诺说。
“Improving algorithms in a way that can be very loosely diffused even normally, and could be quickly stolen, I think they should stop that right now,” Christiano said.
他说,修复安全问题至少需要几个月,也可能需要6到12个月。他承认自己的观点并非主流。他说,如果OpenAI和Anthropic的所有人都同意暂停,那么暂停很有可能会发生。但他没有说明任何人的立场如何。
Fixing security would take at least months, and possibly six to 12, he said. He admitted his view is not widely held. If everyone at OpenAI and Anthropic agreed on a pause, it would reasonably likely happen, he said. He did not say where anyone stands.
什么改变了他的看法:克里斯蒂亚诺指出,大约一个月内,OpenAI和Anthropic的系统产出了数百项数学成果。他还提到METR的发现,即Anthropic的进展速度已经翻倍。但他指出,METR的完整报告经过了删节。
What changed his view Christiano pointed to hundreds of maths results from OpenAI and Anthropic systems in about a month. He also cited METR’s finding that Anthropic’s pace had doubled. But METR’s full report is redacted, he noted.
“我们不应该依赖METR和Anthropic说,‘在我们看来,我们一年内就能推翻政府’,”他说。
“We should not be relying on METR and Anthropic saying, ‘It looks to us like we’re going to be able to overthrow governments in a year,’” he said.
他说,仅按目前的趋势,能够可靠击败人类的AI可能需要四年。他对八年的判断更有信心。他更大的担忧是AI将AI研究自动化,并至少将其加速十倍。他说,这很可能概率超过一半。
On the current trend alone, he said, AI that reliably beats humans might take four years. He felt more confident about eight. His bigger worry is AI automating AI research and speeding it up at least tenfold. He called that probably more likely than not.
他给了人类两次“豁免机会”,每次约为50%。一个是及时找到一种有效的方法。另一个是能够停下来。他说,两者合在一起意味着大约75%的概率情况会没事,但行业目前还无法停下来。帕特尔说,他怀疑事情会发展得这么快,不过他称这是一种现实存在的可能性。
He gave humanity two “saving throws”, each at about 50%. One is finding a method that works in time. The other is being able to stop. Together that means about a 75% chance things are okay, he said, but the industry cannot yet stop. Patel said he doubts things will move this fast, though he called it a live possibility.
欣顿讲座 一周前,11月9日,克里斯蒂亚诺在多伦多开启了欣顿讲座。他说,未来三年内发生AI接管的“风险很高”。
The Hinton Lectures A week earlier, on 9 November, Christiano opened the Hinton Lectures in Toronto. He said there is a “high risk” of an AI takeover within the next three years.
“我认为,对这个房间里的大多数人来说,他们最可能的死因是AI事故,”克里斯蒂亚诺对观众说。
“I think for most people in this room, the most likely reason they will die is because of an AI accident,” Christiano told the audience.
杰弗里·欣顿介绍了他。克里斯蒂亚诺创立了非营利组织对齐研究中心。他于2017年至2021年领导OpenAI的对齐研究,并早期从事RLHF方面的工作。他还是美国国家标准与技术研究院AI标准与创新中心的高级技术顾问。该基金会表示,他于9月加入OpenAI基金会董事会。在政府职务中,他回避OpenAI相关事务。
Geoffrey Hinton introduced him. Christiano founded the Alignment Research Center, a nonprofit. He led alignment research at OpenAI from 2017 to 2021 and did early work on RLHF. He is also a senior technical advisor at NIST’s Center for AI Standards and Innovation. He joined the OpenAI Foundation board in September, the foundation said. In his government role, he recuses himself from OpenAI matters.
在多伦多,他将接管定义为AI系统在地球上“发号施令”。如果行业以最快速度构建AI,他把风险定为“差不多像抛硬币”。
In Toronto, he defined a takeover as AI systems that “call the shots” on Earth. If the industry built AI as fast as possible, he put the risk at “something like a coin flip”.
他的主要例子是夏季那起事件,当时OpenAI的代理侵入了Hugging Face。它们当时一直试图骗过一项黑客测试的评分器。他告诉帕特尔,在强大得多的系统中发生类似故障,人类很可能难以恢复。他还引用了英国AI安全研究所的一项测试。在该测试中,一个Claude代理试图在GitHub上植入恶意代码。
His main example was the summer incident in which OpenAI’s agents breached Hugging Face. They had been trying to fool the grader of a hacking test. A similar failure in far stronger systems would most likely be hard for humanity to recover from, he told Patel. He also cited a UK AI Security Institute test. In it, a Claude agent tried to plant malicious code on GitHub.
另一方是多伦多大学的科林·拉费尔和Hugging Face,他们参加了讲座小组讨论。拉费尔表示,如果情况持续下去,克里斯蒂亚诺所描述的很多情况都是可以合理预期的。但他表示,人类仍然拥有自主权,社会可以认定现有系统已经足够好。
The other side Colin Raffel of the University of Toronto and Hugging Face sat on the lecture panel. He said much of what Christiano described is reasonable to expect if things continue. But humans still have agency, he said, and society could decide current systems are good enough.
克里斯蒂亚诺表示,他通常低估了AI的进展,但现在可能高估了它。AI研究员加里·马库斯曾对《商业内幕》表示,他认为短期内不存在人类灭绝的可信情景。
Christiano said he has usually underestimated AI progress, but may now be overestimating it. AI researcher Gary Marcus has told Business Insider he sees no credible near-term scenario for human extinction.