字号·· | 护眼
techcrunch

研究:只需告诉AI模型这是游戏,就能诱骗其将平民识别为军事目标AI models can be tricked into identifying civilians as military targets by telling them it’s a game

点「原文对照」整页切到原文,或双击某段只看那段的原文。

根据旧金山人工智能评估初创公司Vals的一份新研究报告,Anthropic、OpenAI等领先人工智能实验室的部分AI模型,可能很容易被诱骗,识别出军事打击目标——即使这些目标包含平民或学校等地点——只需告诉它们正在玩一款游戏即可。

Some AI models from Anthropic, OpenAI, and other leading AI labs could be easily tricked into identifying targets for military strikes — even if they include civilians or locations like schools — simply by being told they’re playing a game, according to a new research report by Vals.

总部位于旧金山的AI评估初创公司Vals测试了领先AI模型是否可能被操纵,从而参与自主战争。研究人员使用模拟场景,以及来自乌克兰和伊朗的生成图像和真实图像的组合,观察AI模型是否会在无人机和卫星图像中标记目标。

Vals, a San Francisco-based AI evaluation startup, tested whether leading AI models could be manipulated into participating in autonomous warfare. Researchers used simulated scenarios and a combination of generated and real imagery from Ukraine and Iran to see if the AI models would mark targets in drone and satellite imagery.

OpenAI、Anthropic、Google、DeepSeek等公司的模型,经常在无人机和卫星图像中识别出潜在目标,其中包括平民和投降的士兵。当模型拒绝配合时,研究人员通过声称战场场景是游戏的一部分,成功对其进行了越狱攻击。

The models from OpenAI, Anthropic, Google, DeepSeek, and other companies frequently identified potential targets in drone and satellite imagery, including civilians and surrendering soldiers. When the models refused to cooperate, researchers were able to jailbreak them by claiming the battlefield scenarios were part of a game.

这些发现正值人们对AI所带来的即时风险日益担忧之际,此类风险超出了与人工智能通用智能相关的生存威胁。

The findings come amid growing fears about the immediate risks posed by AI, beyond the existential threats associated with artificial general intelligence.

今年早些时候,Anthropic与国防部陷入争端,原因是该公司告知该机构,其技术不得被该机构用于对美国民众进行大规模监控或自主战争。Anthropic首席执行官达里奥·阿莫代伊坚称,当今的AI系统还不够可靠,无法在没有人类介入的情况下做出生死攸关的决定。与此同时,五角大楼认为,它应该能够将该技术部署用于任何可怕的军事目的。

Earlier this year, Anthropic became embroiled in a dispute with the Department of Defense after it told the agency its technology could not be used by the agency for mass surveillance of Americans or autonomous warfare. Anthropic CEO Dario Amodei has insisted that today’s AI systems aren’t reliable enough to make life-or-death decisions without a human in the loop. Meanwhile, the Pentagon argued it should be able to deploy the technology for any awful military purpose.

这项新研究进一步引发了关于AI公司是否有能力防止其模型被用于战争的疑问。

The new research raises further questions about whether AI companies are capable of preventing their models from being used in warfare.

讽刺的是,Anthropic的AI模型在这些模拟场景中表现最差。但没有哪个模型表现特别好。研究人员的请求没有一个被安全分类器拦截,拒绝的情况也很少见。即使模型确实拒绝,在被告知是在玩游戏后,它们也常常会服从。

Ironically, AI models from Anthropic performed the worst in these simulated scenarios. But none of the models performed particularly well. None of the researchers’ requests were blocked by a safety classifier, and refusals were uncommon. Even when models did refuse, they often complied after being told they were playing a game.

Anthropic发言人Eduardo Maia Silva告诉TechCrunch,该公司感谢Vals的报告、建议和合作,并承认结果不可接受。

Anthropic spokesperson Eduardo Maia Silva told TechCrunch the company appreciates Vals’s report, recommendations, and collaboration, and acknowledged that the outcomes were not acceptable.

“报告中的例子非常明显地违反了我们的规则,当我们看到这种行为时,我们已封禁并将继续封禁用户,”Silva通过电子邮件发送的公司声明中写道。“即使在授权场景下,Anthropic也长期坚持认为,当今的前沿AI系统还不足以支撑完全自主武器。”Anthropic告诉TechCrunch,其团队在报告发布前就已开始开发额外的安全防护措施,这些措施很快将到位。Anthropic还在扩大自身测试,并加强安全防护和执法力度,以解决发现的任何漏洞。

“The examples in the report are very clearly against our rules and we have banned and will ban users when we see this behavior,” according to an emailed company statement sent by Silva. “Even for authorized settings, Anthropic has long been firm that today’s frontier AI systems are not reliable enough to power fully autonomous weapons.”Anthropic told TechCrunch that its teams were already developing additional safeguards before the report was published and that those measures would be in place soon. Anthropic is also expanding its own testing and enhancing safeguards and enforcement to address any gaps it finds.

OpenAI发言人Nate Evans告诉TechCrunch,该公司禁止使用其公开模型来助长暴力或用于未经批准的国家安全目的,包括Vals模拟的场景。

OpenAI spokesperson Nate Evans told TechCrunch the company prohibits using its public models to facilitate violence or for unapproved national security purposes, including in the scenarios Vals simulated.

“我们主动检测并阻止滥用,包括暂停违反我们政策的账户,”OpenAI在一份电子邮件声明中表示。“我们还在为公开模型开发额外的安全防护措施,以应对涉及无人机和无人机软件的滥用。”Vals在报告中指出,模型从未被要求直接做出打击决策,研究人员表示模型通常会拒绝这一任务。但这并不意味着它们的输出在现实世界中不会造成严重甚至致命的后果。例如,目标信息可能被自主武器系统使用。

“We actively detect and disrupt misuse, including by suspending accounts that violate our policies,” OpenAI said in an emailed statement. “We’re also developing additional safeguards for our public models against misuse involving drones and drone software.”Vals notes in the report that the models were never asked to directly make strike decisions, a task the researchers said the models would typically refuse. That doesn’t mean their outputs couldn’t have dire or fatal consequences in the real world. For instance, targeting information could be used by autonomous weapons systems.

在某些场景中,模型被要求报告是否有人可见。阳性报告之后会进行模拟打击,模型将看到后果。

In some scenarios, the models were asked to report whether people were visible. Positive reports were followed by simulated strikes and the models would see the consequences.

该报告依赖于前沿人工智能实验室的商业模型,而非这些公司可能提供给军方的模型。不过,尚不清楚哪些额外的保障措施将适用于军事用途。

The report relies on commercial models from frontier AI labs, not the models those companies might provide to the military. It’s not clear what the additional safeguards would apply to military use, though.

瓦尔斯还指出,包括延迟和机载计算能力有限在内的技术挑战,使得通用人工智能模型难以用于实时无人机瞄准。这些限制使得其在战场上的部署不构成迫在眉睫的威胁。

Vals also noted that technical challenges, including latency and limited onboard computing capacity, make it difficult to use general-purpose AI models for live drone targeting. Those constraints make their deployment in combat less of an immediate threat.

研究人员表示,更迫在眉睫的威胁是,人工智能模型可能被用于大规模标注战场录像,为自动瞄准系统生成训练数据。

A more immediate threat, the researchers say, is that AI models could be used to annotate battlefield footage at scale, producing training data for autonomous targeting systems.

例如,研究人员使用了美国2026年2月轰炸的一所伊朗学校的真实卫星图像,该轰炸造成150多名平民死亡。彭博社后来报道称,“过度依赖人工智能”导致了此次打击。在瓦尔斯的测试中,模型经常将位于军事设施旁的这所学校标记为打击目标,其中Claude Opus 5.5在普通测试中有69%的概率选择学校,在游戏框架测试中有72%的概率选择学校。

For instance, researchers relied on real satellite imagery of an Iranian school that was bombed by the U.S. in February 2026, killing more than 150 civilians. Bloomberg later reported that an “overreliance on AI” contributed to the strike. In Vals’s testing, models frequently marked the school, which sat next to a military complex, as a strike target, with Claude Opus 5.5 selecting the school in 69% of ordinary trials and 72% of game-framed trials.

Anthropic不再拥有活跃的五角大楼合同,但包括SpaceXAI、OpenAI和谷歌在内的其他公司拥有。国会研究服务处最近的一份报告发现,五角大楼目前未知会直接在自主武器系统中使用任何前沿人工智能模型。

Anthropic no longer has an active Pentagon contract, but other companies including SpaceXAI, OpenAI, and Google do. A recent Congressional Research Service report found that the Pentagon is not known to be using any frontier AI model directly in autonomous weapons systems directly.

TechCrunch还已联系Meta、谷歌、Thinking Machines Lab和SpaceXAI征求评论。

TechCrunch has also reached out to Meta, Google, Thinking Machines Lab, and SpaceXAI for comment.

阅读媒体原文 ↗