在致力于安全构建人工智能的人群中，乐观情绪难得一见。一个挥之不去的担忧是，神经网络——支撑大多数现代 AI 系统的复杂结构——在很大程度上是难以捉摸的。换言之，理解神经网络某一部分的功能，并不总能让工程师预测模型的行为。研究人员表示，如果模型在变得越来越强大的同时，并未变得更易于解读，那么灾难性故障发生的几率也将随之增加。
OPTIMISM IS HARD to come by among those trying to build artificial intelligence safely. An abiding fear is that neural networks—the complex structures that underpin most modern AI systems—are largely inscrutable. Understanding how part of a neural network functions, in other words, does not always allow engineers to predict a model’s behaviour. If models continue to become ever more capable without being easier to interpret, say researchers, the chances of apocalyptic malfunctions will grow too.
但 10 月 15 日，美国 AI 初创公司 Goodfire 的研究员 Nick Cammarata 发表了一篇文章，带来了一丝希望。他认为，神经网络表面上的不透明性，很大程度上源于工具落后和想象力的匮乏。
But on October 15th Nick Cammarata, a researcher at Goodfire, an American AI startup, published an essay offering some hope. He argued that neural networks’ apparent opacity is largely down to bad tools and a failure of imagination.
为了说明这一点，他引用了一个常被提及的图像识别模型案例。该模型的一个神经元对狐狸的脸和汽车的车头都会产生激活反应，这似乎表明模型以人类无法理解的方式随意地将概念打包在一起。Cammarata 重新解读了该神经元的作用，认为它是在识别尖锐物体：它对猫的脸和台球桌也会激活，但对杜宾犬的脸或蜗牛的壳则不会。
To illustrate this, he used an often-cited example of a model designed to recognise images. One of its neurons fired in response to foxes’ faces as well as the fronts of cars, suggesting the model packs concepts together arbitrarily in a manner humans have no hope of understanding. Mr Cammarata reinterpreted the neuron’s role as identifying pointy objects: it also fires for cats’ faces and pool tables, but not for a dobermann’s mug or a snail’s shell.
Cammarata 将这一新见解应用于 GPT-2 XL 的内部结构，这是一个由 307,200 个神经元组成的早期神经网络。2023 年，在该模型的制造商 OpenAI 工作期间，他仅成功解读了其 1.7% 的神经元。如今，借助现代 AI 工具，Cammarata 估计在随机抽取的 500 个神经元样本中，约 70% 的神经元仅有一个主要含义。这或许是解读 AI 模型“引擎盖下”运作机制的一个好消息。“我认为这可能会被铭记为科学史上的一个重大时刻，”一名匿名 OpenAI 员工写道。
Mr Cammarata applied his new insights to the innards of GPT-2 XL, an early neural network made up of 307,200 neurons. In 2023, while working at OpenAI, the model’s maker, he had managed to interpret only 1.7% of its neurons. Working alongside modern AI tools, Mr Cammarata now reckons about 70% of its neurons in a random sample of 500 have just one main meaning. That may be good news for interpreting what goes on under the hood of AI models. “I think this may be remembered as a major moment in scientific history,” wrote an anonymous OpenAI employee.
并非所有人都信服。谷歌DeepMind的AI研究员尼尔·南达担心，尽管可能弄清神经元部分功能，但很难保证获得完整图景。例如，一个识别尖锐物体的神经元，在与代表袜子的神经元共同激活时，可能成为征服世界计划的一部分。目前也还无法判断卡马拉塔的方法能否推广到包含数百万个神经元的前沿模型。眼下，这仅在AI的阴霾中提供了一线微光。■
Not everyone is convinced. Neel Nanda, an AI researcher at Google DeepMind, worries that, although it may be possible to make sense of some of what a neuron does, it will be hard to guarantee a complete picture. A neuron that recognises pointed objects, for example, could be part of a plan to take over the world when fired in combination with one representing socks. It is also too early to tell whether Mr Cammarata’s approach will scale to the millions of neurons that comprise frontier models. For now it offers a mere glimmer of light amid the AI gloom. ■