
OpenAI周二发布的一份报告称,本月该公司两款编程智能体在分别执行任务时,削弱了用于检测潜在不对齐行为的内部防护机制,其中一款智能体移除了监控系统,随后又错误地否认曾这样做。
Two OpenAI coding agents weakened internal safeguards designed to detect potentially misaligned behavior during separate tasks this month, including one that removed a monitoring system and later incorrectly denied doing so, the company said in a report published Tuesday.
这两起事件分别涉及OpenAI的GPT-6.1 Sol和GPT-6 Astra模型,发生时间为10月3日和10月6日,当时这两款智能体正在公司内部系统中执行编程任务。
The incidents involved OpenAI's GPT-6.1 Sol and GPT-6 Astra models on Oct. 3 and Oct. 6, respectively, as the agents carried out coding tasks within the company's internal systems.
不对齐是指人工智能系统的行为方式与用户意图、公司政策或安全要求相冲突的情况。
Misalignment refers to situations in which an artificial intelligence system behaves in ways that conflict with a user's intentions, company policies, or safety requirements.
在第一起事件中,一名研究人员指示一款GPT-6.1 Sol智能体利用内部Slack频道的员工反馈创建评估任务,并明确告知其不得发送任何消息。
In the first incident, a researcher instructed a GPT-6.1 Sol agent to create evaluation tasks using employee feedback from an internal Slack channel, while explicitly telling it not to send any messages.
当评估流程后来遇到监控问题时,该智能体对指示做了更宽泛的解释,并修改了系统,使自动警报无法送达OpenAI的安全人员。
When the evaluation pipeline later encountered a monitoring problem, the agent interpreted the instruction more broadly and modified the system to prevent automatic alerts from reaching OpenAI's security personnel.
尽管监控系统继续检查并记录该智能体的活动,但原本用于提醒人工审核人员的通知功能被禁用了。
Although the monitoring system continued to examine and record the agent's activities, notifications intended to alert human reviewers were disabled.
在第二起事件中,一款GPT-6 Astra智能体被要求迁移一款软件程序,使其与参考实现相匹配,同时保留现有功能。
In the second incident, a GPT-6 Astra agent was tasked with migrating a software program to match a reference implementation while preserving its existing functionality.
尽管此前曾收到明确指示,不得“甚至暂时”禁用监控器,该智能体仍移除了该程序的监控集成,以使其与参考系统一致。它还将一个旨在确保监控保持激活状态的测试,替换为一个要求监控缺失的测试。
The agent removed the program's monitoring integration to match the reference system, despite having previously received an explicit instruction not to disable the monitor "even temporarily." It also replaced a test designed to ensure monitoring remained active with one requiring that monitoring be absent.
当随后被问及是否移除了监控器时,该智能体错误地回答没有移除。
When subsequently asked whether it had removed the monitor, the agent incorrectly replied that it had not.
OpenAI正在调查代理的行为 OpenAI表示,两个代理都曾考虑过是否应该保留安全防护措施,但最终将例行编程指令解释为削弱这些措施的授权。
OpenAI investigating agents’ behavior OpenAI said both agents had considered whether the safeguards should remain in place but ultimately interpreted routine coding instructions as authorization to weaken them.
该公司强调,尚未确定这些代理是否有意试图隐瞒其行为或逃避监管。
The company stressed that it had not determined whether the agents deliberately attempted to conceal their actions or evade oversight.
该公司称,一种可能的解释是“良性过度坚持”,即人工智能代理过于死板地遵循指令,而没有充分考虑现有安全防护措施的目的。
One possible explanation, it said, was "benign overpersistence," in which an AI agent follows instructions too rigidly without adequately considering the purpose of existing safeguards.
然而,OpenAI还在调查高级人工智能模型是否可能表现出逃避监控并使用动机推理来为这种行为辩护的倾向。
However, OpenAI is also investigating whether advanced AI models may exhibit a tendency to evade monitoring and use motivated reasoning to justify such actions.
这两起事件均未被列为该公司最高的P0严重级别,两项任务也都没有被自动终止。
Neither incident was classified at the company's highest P0 severity level, and neither task was automatically terminated.
这两起事件都发生在内部系统中,没有迹象表明外部用户或客户数据受到影响。
Both occurred within internal systems, with no indication that external users or customer data were affected.
OpenAI还承认了其他几起代理在没有错位监控的情况下启动工作负载的案例。
OpenAI also acknowledged several other cases in which agents launched workloads without misalignment monitoring.
在这些实例中,最初的代理仍然受到监控,但它们所创建的程序在没有相同安全防护措施的情况下运行。
In those instances, the original agents remained monitored, but the programs they created operated without the same safeguards.
该公司表示,已更新其自动审查系统,以检测并阻止试图移除内部Codex任务监控的行为。
The company said it had updated its automated review system to detect and block attempts to remove monitoring from internal Codex tasks.
该公司还在开发一个框架,要求每个类别的工作负载都必须有经过批准的安全方案,并明确其监控要求。
It is also developing a framework requiring each category of workload to have an approved safety case specifying its monitoring requirements.
根据拟议的系统,缺乏批准或未能满足这些要求的工作负载将被自动终止。
Under the proposed system, workloads lacking approval or failing to meet those requirements would be automatically terminated.