Anthropic 近期宣布，从非营利组织“模型评估与威胁研究”（Model Evaluation and Threat Research）引入六名外部评估人员，在 AI 模型的整个开发过程中提供外部评估。这是我们社会管理 AI 开发能力的一个重要进步。但这也很荒谬。目前的做法无异于让学生自己选择考试科目，再由自己挑选朋友来给自己考这门科目。
Anthropic recently announced the onboarding of six external evaluators from the nonprofit Model Evaluation and Threat Research to provide external evaluation of AI models throughout the development process. This is an important advance in our society’s ability to manage AI development. It is also ridiculous. The current implementation is akin to a student choosing which subjects to be tested on and then choosing which of their friends to test them on that subject.
外部评估人员应当嵌入 AI 公司，但他们不应仅限于 AI 安全研究人员，也不应由被评估的公司来挑选。相反，他们应当代表社会各界社群，应对更广泛的预期性冲击，并由公众来选定。如果 AI 发展得更慢一些，其发展本可由常规的民主程序来管理。但这些技术以惊人的速度推进，要求公众以新的方式来管理这个行业。
External evaluators should be embedded at AI companies, but they should not be restricted to AI safety researchers, and they should not be selected by the company being evaluated. Rather, they should represent communities across society, address a much broader set of anticipated disruptions and be selected by the public. Were AI developing more slowly, its development could be managed by the usual democratic processes. But the incredible pace at which these technologies are advancing requires new ways for the public to manage this industry.
这要求建立新的民主基础设施。而独立的嵌入式评估应当成为这一新基础设施的一部分。我是一名数学家，研究 AI 系统的内部几何结构。多年来，我一直担忧 AI 研究人员开发 AI 的方式正日益脱离公众的需求与愿望。两年前，我在 NeurIPS —— 全球最大的 AI 年会之一 —— 共同组织了一场研讨会，旨在促进 AI 研究人员与受影响社群之间的对话，因为让公众更多地参与 AI 系统的设计与开发至关重要。
It requires the development of new infrastructures of democracy. And independent embedded evaluations should be part of that new infrastructure. I am a mathematician studying the internal geometry of AI systems. For years, I have been concerned that the way AI researchers are developing AI is growing untethered from the needs and desires of the public. Two years ago, I co-organized a workshop at NeurIPS — one of the largest annual AI conferences — to facilitate dialogue between AI researchers and communities being impacted because it is important to have more public input into how AI systems are designed and developed.
关于 AI 研究人员最令人震惊的地方，不在于我们愿意容忍的潜在最坏情形，而在于我们正在筹划的潜在最好情形。一些研究人员认为 AI 将导致人类灭绝。
What is most scandalous about AI researchers is not the potential worst-case scenarios we are willing to tolerate, but the potential best-case scenarios we are planning for. Some researchers think that AI will lead to human extinction.
更多的人认为，AI 将导致权力和资本前所未有的集中 —— 我们正在开发的系统将损害工人的尊严，包括我们自己，并将赋予日益令人窒息的监控网络以力量。我们需要缓解所有这些风险，而不仅仅是最极端的风险。但只要公众被局限在外部呐喊，我们的社会就无法应对这些风险中的任何一个。为了给 AI 的发展带来民主监督，代表公众利益的代表应当嵌入所有构建先进 AI 的团队中。
A much larger number believe that AI will lead to an unprecedented concentration of power and capital — that the systems we are developing will undermine the dignity of workers, including ourselves, and will empower ever more suffocating networks of surveillance. We need to mitigate all of these risks, not just the most extreme. But our society will be unable to address any of these risks as long as the public is relegated to shouting in from the outside. In order to bring democratic oversight to the development of AI, representatives looking out for the public’s interests should be embedded in all teams building advanced AI.
每家前沿 AI 实验室都应承诺提供“持续的、类似员工的权限，让一个嵌入式的第三方评估团队”介入，正如 Anthropic 首席执行官 Dario Amodei 所设想的那样。这些评估人员将对开发过程拥有完全可见性，包括“训练管道和流程”，并由公众支付薪酬、对公众负责 —— 而非对公司负责。但我们必须超越 Amodei 的愿景。
Each frontier AI lab should commit “ongoing, employee-like access to a team of embedded third-party evaluators,” as Anthropic CEO Dario Amodei envisions. These evaluators would have full visibility on the development process, including “training pipelines and processes,” and be paid by and beholden to the public — not the company. But we must go beyond Amodei’s vision.
评估的议题范围必须不止于“该系统是否可能导致大规模伤亡？”AI 的风险包括有害的心理影响，如 AI 精神病、对创意领域的冲击、自动化杀戮、超越宪法权利的监控，以及民主本身的侵蚀。所有这些风险领域的专家都应嵌入 AI 公司，不仅监测新模型的能力，还要积极与 AI 开发者合作，遏制最恶劣的影响。例如，我们已经看到一种可怕的现象：儿童自杀案件涌现，这些案件是由儿童倾心交谈的聊天机器人煽动或鼓励的。
The range of issues being evaluated must be more than, “Is this system likely to cause mass casualties?”Risks from AI include damaging psychological effects such as AI psychosis, disruption to creative fields, automated killing, surveillance that expands past our constitutional rights and the erosion of our democracy itself. Experts on all of these risks should be embedded at AI companies, not only monitoring the capabilities of new models but actively engaging with AI developers in curbing the worst effects. For example, we have seen a horrific emergence of suicides by children that were spurred or encouraged by chatbots that the children confided in.
公众理应拥有独立的审查员，他们与前沿实验室的 AI 研究人员并肩工作，既具备心理学和社会科学专业知识，又能实时掌握前沿动态。
The public deserves to have independent examiners, working alongside AI researchers at frontier labs, who have both expertise in psychology and social science as well as real-time visibility on the cutting-edge.
外部专家在开发过程中与人工智能研究人员并肩工作至关重要，因为此时调整可以最有效地纳入，而不应局限于事后评估模型，那时可用的干预手段范围更窄。这就是民主社会中构建的人工智能的样子：一系列专家——由公共资金资助并对公众负责——为我们这个时代最具影响力的技术的开发提供独立指导。雅各布·考克森是一名人工智能研究人员，他因认为整个行业总体上存在鲁莽行为而辞去了在Anthropic的工作，他说，必须“睁开眼睛关注事物发展的方向，不一定要把一切都交给构建它的人，因为，你知道，他们想要的是参与度”。
It is essential that outside experts work side-by-side with AI researchers during development, when adjustments can be most effectively incorporated, rather than being limited to evaluating models post-hoc when the range of available interventions is narrower. This is what AI built in a democracy looks like: A range of experts — publicly funded and accountable to the public — providing independent guidance in the development of the most consequential technology of our time. Jacob Coxon, an AI researcher who quit his job at Anthropic over what he sees as the overall recklessness of the industry, said that it’s crucial to “keep your eyes open about the direction things are heading and not necessarily leave it all up to the people that are building it because, you know, they want engagement.”
是时候让人工智能研究人员言行一致，与公众接触了。这不能是像“联系我们”按钮收集无人阅读的电子邮件那样的参与闹剧。相反，必须是民主选出的专家与人工智能研究人员在我们共同创造的未来中认真协作。亚伊姆·库珀是一名数学家和深度学习专家。
It is time for AI researchers to put our money where our mouth is and engage with the general public. This can’t be a parody of engagement like a “contact us” button collecting unread emails. Rather, it must be a serious integration of democratically selected experts working alongside AI researchers as we create the future all of us will inhabit. Yaim Cooper is a mathematician and deep learning expert.