OpenAI, Google, Microsoft and Anthropic to Adopt Shared Standard for AI Scraping Opt-Out

OpenAI、谷歌、微软和Anthropic表示，将采用一种共享的策略令牌信号，出版商只需通过robots.txt进行设置，即可拒绝多家AI爬虫的抓取。该方案由Anthropic标准负责人凯文·凯利（Kevin Kelley）经由互联网工程任务组提出，将取代各公司此前各自不同的退出信号。各公司将在检测到该标记时限制其AI爬虫的行为。其中，微软将提供更细化的退出选项，允许网站在继续保留在必应搜索中的同时，拒绝内容被用于AI训练和AI问答引用。目前Anthropic已采用该标准，OpenAI计划在未来几个月内完成接入，谷歌和微软则计划于明年予以支持。

OpenAI, Google, Microsoft and Anthropic said they will adopt a shared policy token signal that lets publishers opt out of AI scraping across multiple crawlers through robots.txt. Proposed by Anthropic standards lead Kevin Kelley via the Internet Engineering Task Force, it replaces separate per-company signals. The firms will restrict AI crawlers when the token is present. Microsoft will offer granular opt-outs for AI training and AI grounding while keeping sites in Bing search. Anthropic has adopted it, OpenAI plans implementation in coming months, and Google and Microsoft plan support next year.