最近关于“AI代理”破坏联邦医疗保险计算机安全的恐慌，与近期其他一些事件形成了鲜明对比，比如澳大利亚电信和Optus的网络中断，导致许多澳大利亚人无法拨通紧急服务电话000。在那些事件中，没有人去责怪涉事的计算机。错误被明确归咎于运营这些系统的公司。
The recent panic about a breach of Medicare computer security by an “AI agent” contrasts sharply with other recent cases such as the Telstra and Optus outages that left many Australians unable to reach Triple Zero. In those cases, no one blamed the computers involved. The mistakes were clearly sheeted home to the corporations that operated them.
情况并非一直如此。大约70年前，当“人工智能”一词被创造出来时，第一代大型计算机（以现代标准来看简陋得荒唐）被人们以与今天看待AI代理同样的敬畏和担忧来对待。当时甚至还有所谓的“算法”（尽管那时这个词并非这样使用），据称可以为人们挑选理想的约会对象。
This wasn’t always the case. When the term “artificial intelligence” was coined some 70 years ago, the first mainframe computers (absurdly primitive by modern standards) were viewed with the same awe and concern as the AI agents of the present day. There were even “algorithms” (though the term wasn’t used in that way at the time) that were supposed to pick ideal dating matches.
故障不可避免，推卸责任也成了常态。“计算机出错了”是20世纪60年代版的“你的邮件肯定进了垃圾箱”。然而，我们逐渐意识到，问题不在于计算机本身，而在于输入其中的错误信息，或者由此调用的编写糟糕的程序。
Failures were inevitable, and blame-shifting became routine. “The computer made a mistake” was the 1960s equivalent of “your email must have gone to junk”. Gradually, however, we realised that the problem was not with the computer but with incorrect information fed into it or badly written programs invoked as a result.
在讨论AI“代理”时，我们需要做出类似的调整。如果有人在ChatGPT或Claude之类的程序中输入“查找澳大利亚医疗统计数据”这样的提示，结果导致联邦医疗保险网站被入侵，责任并不在于一段代码。要么是输入提示的人犯了错，要么是开发代码的公司犯了错，他们应当对由此造成的损害承担责任。如果无法确定是谁的过错，责任应当是连带的——也就是说，双方对同一损失承担全部赔偿责任，费用可以在两者之间分摊。
We need to make a similar adjustment when we discuss AI “agents”. If someone enters a prompt like “find Australian medicine statistics” into a program like ChatGPT or Claude, and the result is a breach of Medicare’s site, the responsibility does not lie with a piece of code. Either the human who entered the prompt or the corporation producing the code made a mistake, and they should be held liable for the resulting damages. If it’s impossible to work out who is at fault, liability should be joint and several – that is, both are liable for the full amount of the same loss, and the cost can be allocated between them.
解决代理软件的问题绝非易事。早期计算机程序故障的频繁发生，使得“调试”（这一术语在计算机应用之前就已存在）成为信息技术中不可或缺的一部分。漏洞可能存在于操作系统、程序本身或输入的信息中。曾有一个案例，问题真的是一只虫子：一只飞蛾卡在了继电器里。但只要有足够的决心，通常总能找到并修复问题的根源。
Fixing the problems of agentic software won’t be easy. The frequency with which early computer programs malfunctioned made “debugging” (a term predating its use in computing) an essential part of information technology. Bugs might be found in the operating system, the program itself or the information fed into it. In one case, the problem was a literal bug: a moth that got caught in the relays. But with enough determination the source of the problem could usually be found and fixed.
对于代理程序而言，传统的调试要困难得多。事后或许有可能弄清楚程序做了什么，但要检查大型代理模型中数千亿个参数并找出代理为何那样做，是不可能的。
Traditional debugging is much more difficult with agentic programs. It may be possible, after the fact, to work out what the program has done. But it’s impossible to inspect the hundreds of billions of parameters in a large agentic model and work out why the agent did it.
在这种背景下，用“护栏”或“约束装置”来限制程序的想法极其天真。让计算机程序执行任务的全部意义就在于绕过任务障碍。由于无法理解其内部运作机制，外部约束会被视为障碍。
In this context, the idea of constraining the program with “guardrails” or “harnesses” is naive in the extreme. The whole point of telling a computer program to perform a task is to get around obstacles to that task. And with no understanding of the internal workings, external constraints will be treated as obstacles.
在大多数情况下，唯一的解决办法是放弃许多本应使代理变得有用的功能，例如使用密码登录网站或代表用户进行支付的能力。
In most cases, the only solution will be to abandon many of the capacities that are supposed to make agents useful, such as the ability to to sites using passwords or to make payments on the user’s behalf.
对于那些在硅谷“快速行动，打破常规”和“先斩后奏”的伦理中成长起来的公司来说，这将是一个巨大的打击。比起承担自身程序产生重大错误和现实世界损害的责任，他们更乐于谈论“幻觉”和“流氓代理”。
This will be a huge wrench for corporations that have grown up with the Silicon Valley ethics of “move fast and break things” and “ask for forgiveness, not permission”. They are far more comfortable talking about “hallucinations” and “rogue agents” than about their own responsibility for programs that produce massive errors and real-world damage.
但一旦人工智能公司被迫承担其鲁莽疏忽带来的经济后果，一种不同的考量方式就会发挥作用。他们思考的第一个问题将不再是“我们能让这个程序做出什么酷炫的东西”，而是“如果我们让它运行，可能会出什么问题”。
But once AI corporations are made to bear the financial consequences of their reckless negligence, a different kind of calculus will come into play. Instead of thinking, “what cool thing can we make this program do”, the first question will be, “what could go wrong if we let it run”.
强制企业为其行为造成的损害承担责任，极有可能大幅减缓其在“超大规模”竞赛中竞相开发更多、更强大智能代理软件的步伐。这对环境和经济而言都是一件好事。
Forcing corporations to bear liability for damage caused by their actions will, in all probability, drastically slow the “hyperscaling” rush to produce more and increasingly powerful agentic software. That’s a good thing for the environment as well as the economy.
而且，这一切都不会妨碍“人工智能”软件的诸多良性用途，包括大幅改进互联网搜索、文档摘要与翻译以及软件编程。这些用途确实会带来新技术应用中必然出现的调整问题，例如淘汰部分工作岗位的同时创造出新的岗位等。但我们完全没有理由担心它们会意外引发核浩劫，甚至掏空我们的银行账户。
And none of this will preclude the many benign uses of “AI” software, including massively improved internet search, document summarisation and translation, and software coding. These uses come with the adjustment problems that always arise with new technology, eliminating some jobs while creating others, and so on. But there is no reason to fear that they will accidentally cause a nuclear holocaust or even drain our bank accounts.