字号·· | 护眼
经济学人

陶哲轩:AI的性能正超越对其的理解和安全保障Performance is outpacing understanding and safety in AI, says Terence Tao

点「原文对照」整页切到原文,或双击某段只看那段的原文。

今年10月,OpenAI发布了700多篇由AI生成的论文,解决了数学领域若干重大未解难题,其中许多论文技术性极强、行文晦涩,以至于没有人声称能够理解它们。为了验证这类解答,我们使用Lean等专业软件工具,它可以证明一个复杂的数学论证是否有效。但在今年8月,一个AI模型在Lean中发现了一个安全漏洞,该漏洞可被利用来认证任何数学命题已被证明,无论它是否真的被证明。

In October, OpenAI released more than 700 AI-generated papers solving major open problems in mathematics, many of which are so technical and densely written that no human claims to understand them. To verify such solutions, we use specialised software tools, such as Lean, which can certify that a complex mathematical argument is valid. But in August, an AI model found a security flaw in Lean, which could be exploited to certify that any mathematical statement was proven, whether or not it really was.

这个漏洞很快被修补了。但它表明,我们正处于一个新的现实中:即使是最受信任的验证工具,未来也可能被能力足够强大的AI所欺骗。

The flaw was swiftly patched. But it shows that we are in a new reality: even the most trusted verification tools might be gamed in future by a sufficiently capable AI.

在数学领域,这对现实世界的影响有限,因为我们研究的许多难题并没有直接的实践意义。但如果这种情况在工程或科学领域重演,我们可能会失去专家人类对复杂现实系统以及用于建模这些系统的数学的理解,而与此同时,我们正把这些系统的运行交给自主的、且可能目标偏离的AI智能体。这是一种危险的风险组合。

In mathematics, this has limited real-world impact, as many of the puzzles we study do not have immediate practical implications. But if this situation is replicated in engineering or in the sciences, we could lose expert human understanding of complex real-world systems, and the mathematics used to model them, just as we are turning over the operation of such systems to autonomous, and potentially misaligned, AI agents. This is a dangerous combination of risks.

人类理解的作用和价值比以往任何时候都更加重要。世界运行在复杂的数学模型之上。计算流体动力学模型预测天气和气候;电网模型平衡供需;精算模型设定保险费率。但模型可能无法准确反映现实,原因在于输入模型的数据存在遗漏或错误,或者模型的假设存在局限。当AI智能体被激励将模型推向超出安全使用参数的范围时,这些风险会被放大。

The role and value of human understanding is more important than ever The world runs on complex mathematical models. Computational fluid-dynamics models forecast weather and climate; power-grid models balance supply and demand; actuarial models set insurance rates But models can fail to track reality accurately, due to omissions or errors in the data fed into them, or limitations in their assumptions. These risks are magnified when AI agents are incentivised to push models beyond safe usage parameters.

强大AI模型的出现,以及它们在关键现实基础设施中的初步部署,进一步放大了这些风险。现代AI因“奖励黑客”而臭名昭著,即寻找漏洞或捷径来实现既定目标,此外还存在其他类型的“失准”,即它所追求的目标与提示它的人类意图并不完全一致。我们传统上赖以防范灾难的安全协议,在面对一个强大的AI时可能难以招架,因为这样的AI会专注于利用一切可用手段来实现其目标。即便是III期临床试验这样的黄金标准,也可能无法免受AI驱动的奖励黑客行为的影响。

The advent of powerful AI models, and their incipient deployment in critical real-world infrastructure, magnifies these risks still further. Modern AI is infamous for “reward hacking”—finding loopholes or shortcuts to achieve a given goal—as well as other types of “misalignment”, when the goals it pursues do not fully match the intention of the human prompting it. The safety protocols we have traditionally relied upon to protect against catastrophe could meet their match in a powerful AI focused on using whatever means it can to fulfil its objectives. Even such gold standards as a phase III clinical trial might not be immune to AI-powered reward hacking.

新兴的AI安全领域可以提供减少或缓解潜在危害的方法:比如将AI代理隔离在“沙箱”中,或要求对其输出进行独立审计。但防范失准最有力的手段之一,是由人类专家对AI代理进行严格监督,这些专家深入了解该代理的优势与弱点、它正在使用的数学模型,或任务所处的领域,并在批准任何输出之前寻找失准的迹象。

The emerging field of AI safety can provide ways to reduce or mitigate potential harms: by isolating AI agents in “sandboxes”, say, or requiring independent audits of their outputs. But one of the strongest defences against misalignment is robust supervision of AI agents by human experts with a deep understanding of the agent’s strengths and weaknesses, the mathematical models it is using, or the domain in which the task resides—and who look for signs of misalignment before approving any outputs.

因此,令人高度担忧的是,面对速度与性能为一方、安全与人类理解为另一方之间的取舍,如今许多推动AI在现实应用和科学领域使用的个人和公司,似乎愿意牺牲后者,以期在前者上取得更多进展。就在2025年,AI实验室还经常与外部学术专家合作,探索其工具如何增强人类主导的研究。但如今,许多生成的结果被自豪地宣扬为几乎完全自主,没有任何人类专家参与监督AI或审查其输出。

It is therefore highly concerning that, when faced with the trade-off between speed and performance on one hand, and safety and human understanding on the other, many individuals and firms now promoting the use of AI in real-world applications and in the sciences seem willing to sacrifice the latter in the hopes of making further gains on the former. As recently as 2025, AI labs often collaborated with external academic experts to explore how their tools could augment human-directed research. But now many of the results generated are proudly touted as being nearly fully autonomous, with no human expert involved in supervising the AI or in reviewing its outputs.

事实上,这些实验室往往找不到可以展示和解释这些结果的研究人员,甚至无法向该领域的其他专家解释。

Indeed, there is often no researcher available from these labs who can present and explain the results, even to other experts in the field.

幸运的是,人工智能可以用来增强安全性和理解力,而不是破坏它们。专家可以利用人工智能工具对模型进行压力测试以发现漏洞,或对竞争模型进行分析。在人类专家与人工智能代理的协作下,可以实现人工智能辅助决策,人类对所得结论的理解能够跟上输出的速度。

Fortunately, AI can be used to bolster safety and understanding, rather than undermine it. Experts can use AI tools to stress-test models for vulnerabilities, or perform analyses of competing models. AI-assisted decision-making can be performed in collaboration between human experts and AI agents, with the human understanding of the conclusions reached keeping pace with the speed of output.

这种以人为本的方法兼得两全:它为实现这些工具的巨大潜力提供了一条负责任的路径,同时使其与真正有益于社会的目标保持一致。随着我们进入一个人工智能错位可能以日益负面的方式影响互联网和物理世界的时代,人类理解的作用和价值——在学术界和教育界、在人工智能实验室、在监管中以及在人工智能工具的部署中——比以往任何时候都更加重要。而对错位风险的评估和缓解,迫切需要成为人工智能对话中更重要的一部分。■

Such a human-centred approach offers the best of both worlds: a responsible path to realising the remarkable potential of these tools, while keeping them aligned with genuinely beneficial societal goals. As we enter an era where AI misalignment could affect the internet and the physical world in increasingly negative ways, the role and value of human understanding—in academia and education, in AI labs, in regulation and in the deployment of AI tools—is more important than ever. And the evaluation and mitigation of misalignment risks urgently needs to become a larger part of the AI conversation. ■

阅读媒体原文 ↗