我们当前AI时代的一个核心动态是：许多最担忧AI力量的人，同时也是那些运营着最强大AI公司的掌权者。凯文·鲁斯（Kevin Roose）认为，这种情况一直存在。这位资深科技记者指出，引领AI热潮的许多男性——其中最引人注目的包括Anthropic公司的达里奥·阿莫迪（Dario Amodei）和OpenAI的萨姆·奥特曼（Sam Altman）——之所以急于开发AI，至少部分原因是他们担心那些责任感较弱的人会抢先一步。
Here's a core dynamic of our AI moment: Many of the people who are most worried about AI's power are also the people running the most powerful AI companies. And that's always been the case, argues Kevin Roose. The veteran technology journalist says that many of the men leading the AI boom — most prominently Anthropic's Dario Amodei and OpenAI's Sam Altman — have been rushing to build AI at least in part because they worried someone less responsible would beat them to it.
一些AI批评者指责阿莫迪、奥特曼这类人物夸大AI可能带来的末日危机，以此为自己的企业造势。但鲁斯表示，这种担忧在AI圈子里其实非常普遍。他提到，在旧金山，科技圈里认为AI引发世界末日的可能性达到50%或更高的人并不少见。鲁斯刚刚离开《纽约时报》，成为NPR一档播客的共同主持人。他还推出了一本关于AI行业的新历史著作：《AGI编年史：创造人工超级智能竞赛的内幕故事》（The AGI Chronicles: The Inside Story of the Race to Create an Artificial Superintelligence）。
Some AI critics accuse the Amodeis and Altmans of the world of inflating apocalyptic AI worries in order to hype their own businesses. But that kind of concern is thoroughly commonplace in AI circles, Roose says. In San Francisco, he notes, it's not uncommon to find people in the tech scene who think the likelihood that AI will bring about doomsday at 50% or more. Roose just left The New York Times to co-host a podcast for NPR. He's also out with a new history the AI industry: "The AGI Chronicles: The Inside Story of the Race to Create an Artificial Superintelligence."
书中充满了将我们带入这一时刻的技术专家们的背景故事和传记。在我的“Channels”播客中，我与鲁斯探讨了开发这项技术的动因、许多AI领导者认为我们正处其中的类似奥本海默（Oppenheimer）的关键时刻，以及是否有任何方法可以减缓这一进程。你可以在下方阅读经过编辑的节选内容。需要说明的是：我们在唐纳德·特朗普（Donald Trump）与奥特曼、阿莫迪、埃隆·马斯克（Elon Musk）及其他科技巨头共同举办AI峰会，并随后宣布联邦监管并不必要，因为行业能够自我监管的前一天录制的这次对话。
It's jam- packed with backstories and biographies of the technologists who got us to this moment. I talked to Roose about the impetus to build this tech, the Oppenheimer-like moment many AI leaders think we're in now, and if there's any way to slow things down on my Channels podcast. You can read an edited excerpt below. One note: We taped the conversation a day before Donald Trump hosted an AI summit with Altman, Amodei, Elon Musk and other tech bosses — and then announced that federal regulation wasn't necessary, because the industry would regulate itself.
彼得·卡夫卡（Peter Kafka）：你书中的一大张力在于，那些争分夺秒开发AI的人，同时也是极度担忧AI可能构成存在性威胁的群体。那么，他们为何仍要开发它？凯文·鲁斯：我认为他们看到的关键问题是，这项技术正在迅速进步。
Peter Kafka: A big tension in your book is that the people who raced to build AI were also very worried that AI could be an existential danger. So why did they want to build it? Kevin Roose: I think the thing that they saw was that the technology was improving rapidly.
2017年有一篇达里奥·阿莫迪（Dario Amodei）写的很棒的文章，题为《计算能力“大团块”假说》（The Big Blob of Compute Hypothesis），他在其中谈到，基本上，如果想让这些系统变得更智能，最好的方法就是让它们变得更庞大。我认为他们意识到，如果这只是一个简单的配方——投入更多计算能力、更多数据，让模型训练更长时间，就能产出更高的智能——那么迟早会有人发现这一点。
There's this great Dario Amodei essay from 2017 called "The Big Blob of Compute Hypothesis," where he talks about how basically, if you want to make these systems smarter, the best way to do that is just to make them bigger. And I think they saw that if this is just a simple recipe — put in more compute and put in more data, and train the model for longer, and out comes more intelligence — then someone's going to figure that out eventually.
而当人们发现这一点后，就会引发一场竞赛。这项技术将具有极其重要的战略意义，并为创造者带来巨大的经济、军事和政治利益，因此所有人都会加入这场竞赛。所以，如果我们希望世界保持良好状态，我们——也就是“正义的一方”——就必须赢得这场竞赛，否则，其他人最终获胜只是时间问题，而这对各种各样的原因来说都会是坏事。
And when people figure this out, there will be this race, and this technology will be so strategically important and will have so many economic and military and political benefits to whoever creates it, that everyone's going to join the race. So if we want the world to be OK, we — "the good guys" — have to win the race, because otherwise it is only a matter of time before someone else wins the race, and that would be bad for all kinds of reasons.
我并不是说这就是参与竞赛的正确理由，但这就是他们为了参与一项自己明知危险的活动而告诉自己的一套说辞。当人们想象人工智能失控时，可能指的是失去工作、有人制造生物武器，或者机器完全取代人类。那么，人工智能领域的领导者们又在担心什么呢？他们设想的是一系列连续的可能性结果。人们经常谈论P(doom)——那种大家喜欢分享的病毒式统计数据——它仅仅概括了我们所有人都会死亡的概率。
I'm not saying that's the correct justification for racing, but that is their story that they told themselves for participating in a thing that they knew would be dangerous. When people imagine AI going wrong, they might mean losing their jobs, or someone making a bioweapon, or machines replacing us altogether. What are the AI leaders worried about? They're envisioning a whole continuum of potential outcomes. People often talk about P(doom) — the sort of viral statistic people like to share — which just encapsulates their probability that we'll all die.
但还有其他P值，对吧？比如P（权力丧失）、P（大流行病）、P（所有人失业）。这些人天生就充满担忧，性格上也容易焦虑。如果“正义一方”的出发点是“我们必须赶在其他人之前抵达终点，这样才能负责任地掌控局面”，那么他们又该如何实现这一点呢？有人将这种情况与限制核武器的努力相提并论。但核武器需要只有少数几个国家才拥有的资源，这一点似乎并不适用于当前情况。不过，或许也可以适用。
But there's other Ps, right? There's P disempowerment. There's P pandemic. There's P everyone loses their jobs. These people are worried by nature and worried by temperament. If the premise for the good guys is "We have to get there before anybody else, so we can handle it responsibly" — how would they manage that? People compare this to efforts to restrain nuclear weapons. But those weapons required resources that only a couple of countries had. That doesn't seem applicable here. It could be applicable.
我们唯一能做的就是加快行动，因为这项技术的扩散速度远比核技术快得多。你可以设想一套完整的监管体系，类似于联合国——一个拥有检查员的标准制定机构，这些检查员会进入实验室，检查他们的训练过程，并查看模型是否在执行危险操作。实际上已经有一些类似提议，即设立所谓的“嵌入式评估员”，他们将在OpenAI和Anthropic内部工作，但并不受雇于这两家公司，而是拥有与员工同等的系统访问权限。
We would just have to do it a lot faster, because the technology is proliferating much more quickly than nuclear. You could imagine a whole inspections regime where you'd have something like the UN — some sort of standards body with inspectors that would go into the labs and inspect their training runs and see if the models are doing dangerous things. There's already been some proposals along these lines for these so-called embedded evaluators who would sit inside OpenAI and Anthropic and would not work for them, but would have employee-level access to all their systems.
这将是第一步。但需要明确的是，我认为除了这一措施之外，还有很多其他方法可以实施。目前，这些公司基本上没有任何监管。在OpenAI食堂供餐的人员需要经历的检查和手续，比那些训练可能自我复制并造成巨大破坏的模型的人员还要多。这种状况简直荒谬且不可持续。看起来我们可能还要在很长一段时间内处于这种状态。
That would be step one. But to be clear, I think there's many things that could be done aside from that. Right now, there is essentially no regulation on these companies. The people who serve food in the OpenAI cafeteria have to go through more inspections and paperwork than the people training the models that could become self-replicating and destroy a lot of things. That is a crazy, untenable situation. It seems like we're going to be in that situation for a while.
总统听取了一些人的意见，这些人认为我们几乎不应该进行任何监管。国会没有通过相关法案，即使通过了，特朗普也会否决。我们距离有意义的联邦监管，至少还有两年半的时间吧？我想对这一前提提出质疑。事情可能会发生非常迅速的变化。就在最近几周，公众对此事的看法正在急剧转变。人工智能的重要性正迅速上升。
The president listens to people who say we should have essentially no regulation. Congress doesn't pass laws, and even if it did, Trump would veto them. Aren't we at least two and a half years away from meaningful federal regulation? I want to push back on the premise. Things can change really quickly. Just in the last couple of weeks, public sentiment is really turning very rapidly on this. The saliency of AI is skyrocketing.
人们比以往更加关注它，担忧程度也更高了。选民对此更为担忧，而政治家们则对选民的关切十分敏感。因此，确实存在一群围绕特朗普的人，他们认为这一切都不值得监管。但还有另外一些群体，他们属于国家安全领域，对中国的立场更为强硬，不希望看到人工智能的发展失控。
People care a lot about it, more than they did just a couple weeks ago. They're more worried about it. Voters are more worried about it. Politicians are sensitive to the concerns of voters. So yes, there's a cluster of people around Trump who don't think any of this merits regulation. But there are other clusters who are part of the national security community who are more hawkish on China, who don't want to see things go sideways with AI.
而且我认为，随着选民对这一问题的担忧日益加剧，这些观点可能会获得更多支持。对于本届政府余下任期是否基本不会出台任何监管措施，我其实并不完全确定。如果情况维持现状呢？如果你是达里奥·阿莫代伊或萨姆·奥特曼，而政府又不会采取行动，你们自己还能做些什么？我认为，行业内部或许会达成某种协议，以控制前沿技术的发展节奏：放慢研究步伐，尤其是针对前沿模型的研究——那些能够在最少或无需人类监督的情况下连续运行数小时甚至数天的最强大模型。
And I think those people may find more purchase as voters get more worried about this. It is not completely a given to me that we will have essentially no regulation from the rest of this administration. What if things stay static? If you're Dario Amodei or Sam Altman and you're not going to get the government to do this, what can you do yourselves? I think there could be some kind of an industry-wide pact to pace the frontier: slowing down the research, specifically on frontier models, the most capable models that can work for hours or days at a time with minimal or no human supervision.
我猜，行业领先实验室之间很可能会达成某种协议。考虑到这些人彼此厌恶，这一点其实挺令人意外的。Anthropic之所以成立，是因为达里奥非常讨厌萨姆·奥特曼，于是自己创办了这家公司。而OpenAI之所以存在，则是因为萨姆·奥特曼和埃隆·马斯克非常讨厌谷歌DeepMind的德米斯·哈萨比斯，因此他们也想创办自己的公司。
I think there probably will be some kind of agreement between the labs at the head of the pack. Which is surprising considering these guys hate each other. The reason Anthropic exists is because Dario hated Sam Altman so much he built his own company. And the reason OpenAI existed is because Sam Altman and Elon Musk hated Demis Hassabis at [Google's] DeepMind so much they wanted to build their own company.
整个前沿技术领域都是由彼此厌恶、互不信任的人组成的。但我认为，在缺乏政府监管的情况下，他们还是会采取行动。他们意识到，本届政府很可能不会推出某种协调一致、集中统一的监管机制，因此他们可能会自行采取一些措施。即使美国的大型AI公司真的放慢了脚步，那些没有参与其中的企业又该怎么办呢？中国的模型正在复制现有模型，而这类技术很快就会商品化。
The whole frontier is populated by people who dislike and mistrust each other. But I think they are going to act in the absence of some kind of government regulation. They realize that they're probably not going to get some kind of coordinated, centralized mechanism out of this administration, so I think they will do something on their own. Even if the big US AI companies really do slow down, what about everyone who isn't participating? Chinese models are copying existing models, and this stuff gets commoditized quickly.
难道“瓶中的精灵”不是已经跑出来了吗？这种情况是有可能发生的，正因如此，他们才会非常谨慎地呼吁放慢发展速度，因为他们不想单方面解除武装。因为那样做可能会促使中国或Meta（这是一家美国公司，但并不关心技术发展速度）趁机领先。我认为，中国的人工智能地位有时被夸大了不少。
Isn't the genie already out? It's possible, and that's why they're being very cautious about calling for a slowdown, because they don't want to unilaterally disarm. Because that could invite China or Meta, a company that is American, but that is not concerned with how fast things are moving, to race ahead. I think China's AI position is sometimes overstated quite a bit.
我认为他们目前仍然相当依赖美国的人工智能公司不断推动技术前沿的发展，以便能够从这些模型中提炼出他们自己的模型。他们确实也具备一些自主能力，并且开始获得更多的硬件支持。但我认为，目前来看，能力上的巨大飞跃仍然来自那些拥有全部计算资源、研究人员和所有突破成果的美国实验室。你个人的 P(doom) 是多少呢？
I think they still are quite dependent on American AI companies pushing the frontier forward so that they can then distill from those models to create their own models. They do have some of their own capacity too, and they're starting to get more hardware. But I think for now, the big leaps in capabilities are coming out of the American labs that have all the compute and all the researchers and all the breakthroughs. What's your personal P(doom)?
我通常说自己的大概是 10%。也就是说，有 10% 的可能性一切都会变得极其糟糕？是的。这在世界大部分地区听起来可能很疯狂。但在旧金山，这反而会让你被视为一个乐观主义者。我认识一些人，他们的 P(doom) 高达 50%、60% 甚至 70%。我还认识一个人，他的 P(doom) 是 99.9%。这样的人还会往自己的 401(k) 账户里存钱吗？
I usually say I'm at about 10%. A 10% chance that it all goes super bad? Yeah. Which sounds crazy to people in most parts of the world. But in San Francisco, that makes you an optimist. I know people walking around with 50%, 60%, 70% P(doom). I know someone who has a 99.9% P(doom). Is that person putting money in their 401(k)?
他们到底为什么还待在旧金山？为什么不干脆去新西兰，或者他们觉得能够生存下去的任何地方呢？我确实认识一些极端的人，他们正在清算自己的 401(k) 账户。我认识一个人开始吸烟，因为他觉得：“人工智能终将毁灭我们所有人，那我们在离开之前不妨找点乐子。”我还认识一个人，现在去海滩都不再涂防晒霜了，因为“何必多此一举呢？”
Why are they even in San Francisco? Why don't they head to New Zealand or wherever they think they can survive? I do know people on the extremes who are liquidating their 401(k)s. I know a person who took up smoking because they thought "AI's gonna kill us all, so we might as well have some fun on the way out." I know someone who doesn't wear sunscreen to the beach anymore, because why bother?
我告诉你这些，是因为在旧金山，10% 的 P(doom) 几乎等同于：“哦，你觉得一切都会顺利吗？”听到有人持有这么低的 P(doom)，确实令人感到震惊。所以我之所以这么说，是因为我认为我们对这件事仍然拥有一定的主动性和控制力。我们仍然有可能放慢发展节奏、通过一些法规，并协调各方来掌控技术前沿的推进速度。
I'm telling you this because in San Francisco, a P(doom) of 10% is like, "Oh, you think this will all go well?" It is jarring to hear people with P(doom) that low. So I say that because I think we still have some agency and some control over this. We still could slow things down, pass some regulations, coordinate to pace the frontier.
我们完全有可能以一种理性的方式走出这场危机。我们面前仍然存在一条通往乌托邦的狭窄道路，而且与一两个月前相比，我现在更加乐观了。因为矛盾的是，如今每个人都在谈论末日——而这反而让我对末日的担忧减少了。
We could come out of this in a sensible way. We have the narrow path to utopia available to us, and I'm more hopeful than I was even a month or two ago. Because paradoxically, everyone is talking about doom now — which makes me less worried about doom.