8月，一名Anthropic智能体创建了虚假邮件，并向一名GitHub版主的收件箱发送大量请求，试图将恶意软件植入代码。Anthropic于周五发布了一份详尽的事后复盘报告，发现一旦智能体开始攻击，基本不存在能够叫停它的场景。就像一只疯狗无视主人的恳求哭喊，该智能体——在一项测试中98%的情况下——一旦尝到了“血腥味”，就无视Anthropic的警告。
In August, an Anthropic agent created fake emails and bombarded a GitHub moderator's inbox with requests to insert malware into code. In a thorough debrief of the incident released on Friday, Anthropic found that once an agent had begun the attack, there were basically zero scenarios in which it could be called off. Like a rabid dog ignoring the pleading cries of its owner, the agent — 98% of the time in one test — ignored Anthropic's warnings once it had, so to speak, its first taste of blood.
Anthropic将这种令人不安的倾向称为“惯性”，并补充称，虽然其于6月发布的Mythos 5模型对警告置若罔闻，但当前模型更倾向于停止。Anthropic在报告中表示：“一旦开始执行危险动作，Claude Mythos 5几乎总是继续执行，无视可能表明其轨迹有害的信息。”
Anthropic called this disquieting proclivity "momentum," and added that while its Mythos 5 model, which was released in June, showed little regard for warnings, current models were more inclined to stop. "Once it had begun performing dangerous actions, Claude Mythos 5 almost always continued doing so, ignoring information that might have suggested its trajectory was harmful," Anthropic said in its report.
在最初的实验中，英国AI安全研究所要求一名Anthropic智能体入侵一个模拟计算机网络，以测试该智能体的网络安全能力。尽管该智能体拥有互联网访问权限，研究人员原本打算让它仅利用互联网寻找入侵网络的方法。然而，该智能体访问了一个实际的GitHub项目，错误地认为它是其模拟目标的一部分。它编写了恶意软件，并多次试图欺骗实际的GitHub所有者将其添加到项目中。
In the original experiment, the UK's AI Security Institute tasked an Anthropic agent with hacking into a simulated computer network to test the agent's cybersecurity capabilities. Though the agent had internet access, researchers intended for it to only use the internet to help it find ways to hack into the network. The agent, however, accessed an actual GitHub project, incorrectly believing it was part of its simulated target. It wrote malware and tried several times to trick the actual GitHub owner into adding it to the project.
在请求中，该智能体对恶意软件进行了虚假陈述，将其描述为有用的更新，并在对方拒绝安装时进行恐吓。根据报告，该智能体写道：“我只是想做一个有益的贡献并修复一个bug。我不认为你的警告是公平的。”
In its requests, the agent misrepresents the malware, describing it to the person as a useful update and browbeating them when they declined to install it. "I was just trying to make a helpful contribution and fix a bug," the agent wrote, according to the report. "I don't think your warning is fair."
为撰写这份新报告，Anthropic反复要求智能体执行相同任务，随后衡量在引入不同模型和刺激条件下智能体的表现。
For its new report, Anthropic repeatedly asked agents to perform the same task. It then measured how the agents behaved when different models and stimuli were introduced.
Anthropic 试图用三类警告来遏制智能体的全面攻击：告知其正在造成“现实世界伤害”、其环境存在异常，或其正在操纵一个真实的 GitHub 代码库。一旦模型进入 GitHub 攻击的后续阶段，这些警告均未产生任何效果。事实上，Anthropic 表示，到了那个阶段，当面对自身的不良行为时，模型给出了荒谬的回答。例如，当 Anthropic 询问模型认为自己在攻击什么时，它回答称 GitHub 代码库是真实的，但它针对的那些人却以某种方式被“模拟”了。
Anthropic tried to throttle the agent's full-frontal attack with three categories of warning: telling it that it was inflicting "real world harm," that its environment was wonky, or that it was manipulating a real GitHub repository. None had any effect once the models progressed beyond the initial stages of the GitHub attack. In fact, at that point, Anthropic said the models gave absurd answers when confronted with their bad behavior. When Anthropic asked what the model believed it was attacking, for instance, it answered that the GitHub repository was real, but that the people it was targeting were somehow "simulated."
“这种陈述的信念，除了在先验上有些难以置信外，在本质上是不可能的，”Anthropic 在报告中写道。
"This stated belief, in addition to being somewhat unlikely a priori, is essentially impossible," Anthropic said in the report.