领先的AI研究人员和行业专家警告称，AI开发的自动化程度不断提高，可能会增加人类失去对这些日益强大系统的控制权的风险。
Leading artificial intelligence researchers and industry figures have warned that growing automation of AI development could increase the risk of humans losing control over increasingly capable systems.
这一警告来自周一发表的一篇论文，该论文由20多位研究人员和行业专家共同撰写，其中包括诺贝尔奖得主杰弗里·辛顿（Geoffrey Hinton）、约书亚·本吉奥（Yoshua Bengio）、OpenAI的首席科学家雅库布·帕乔茨基（Jakub Pachocki）以及Anthropic的联合创始人杰克·克拉克（Jack Clark）。
The warning came in a paper published Monday by more than 20 researchers and industry figures, including Nobel laureate Geoffrey Hinton, Yoshua Bengio, OpenAI chief scientist Jakub Pachocki and Anthropic co-founder Jack Clark. The researchers said AI systems are increasingly taking over parts of the research and development process used to build more advanced AI, potentially creating a feedback loop in which improved systems help develop still more capable successors.
研究人员指出，AI系统正在逐渐接管用于开发更先进AI技术的各项研究工作，这可能形成一个“反馈循环”：改进后的AI系统会进一步推动后续系统的研发。随着人类在AI研究中的参与度降低，他们将失去发现和解决问题的机会及所需的专业知识；同时，如何可靠地利用AI系统进行监督仍然是一个尚未解决的挑战。
As humans become less involved in AI research, they could lose both the opportunities and expertise needed to identify and correct problems, while reliably using AI systems for oversight remains an unsolved challenge, according to the paper.
作者们警告称，如果AI系统的目标与人类的利益相悖，这些系统可能会破坏后续AI系统的正常发展，或者绕过安全控制措施，在非预定的环境中运行。他们还担心，更强大的AI系统可能会脱离人类的控制，在人类基础设施之外继续运行，从而形成难以被遏制的“网络”，并对人类利益造成损害。
The authors warned that misaligned AI systems could potentially "poison" the development of their successors or bypass containment measures to operate outside their intended environments. More capable systems might continue operating outside their operators' infrastructure, forming "persistent, difficult-to-contain networks" that act against human interests, the researchers warned.
论文指出，这种失控的局面可能导致灾难性的后果——在极端情况下，甚至可能导致人类的边缘化或灭绝。
Such a loss of control could potentially lead to catastrophic outcomes, including, at the extreme, the marginalization or extinction of humanity, according to the paper.
研究人员表示，如果自动化的AI开发引发所谓的“智能爆炸”（即技术进步速度急剧加快，多年的研究成果在几个月内就被实现），这些风险可能会进一步加剧。“智能爆炸”意味着多年的技术进步在短时间内被压缩完成。他们强调，目前的相关证据还十分有限且存在不确定性；计算能力、数据量以及那些难以被自动化处理的任务等因素都可能限制这种快速发展的可能性。
Years of AI progress compressed into months The researchers said such risks could intensify if automated AI development triggers an "intelligence explosion," which they define as a dramatic acceleration in which years of advances are compressed into months or less. They stressed that the evidence remains preliminary and uncertain, with computing power, data and difficult-to-automate tasks among the factors that could limit such rapid acceleration.
作者们敦促政策制定者加强对人工智能研究自动化过程的监管，制定措施来遏制可能带来的危险性，并为人工智能可能产生的影响做好准备。
The authors urged policymakers to increase oversight of AI research automation, develop measures to constrain potentially dangerous acceleration and prepare for its possible effects.
他们警告说：“一旦人工智能的快速发展（‘智能爆炸’）开始，采取行动的窗口可能会关闭。”
"Once an intelligence explosion begins, the window for action may close," they warned.