随着越来越多的人借助AI聊天机器人倾诉内心最痛苦的思绪，各公司正面临着关键抉择：该如何让自家平台在用户遭遇心理健康危机时提供有效帮助？根据Scale AI开展并独家提供给《时代》周刊的最新研究，这些聊天机器人虽然能够准确识别出用户正处于危机状态，却常常无法提供足够的援助。
As more people turn to AI chatbots to share their most distressing thoughts, companies are making pivotal decisions about how their platforms respond to users in the midst of mental-health crises. According to new research conducted by Scale AI and shared exclusively with TIME, chatbots are adept at discerning when a user is in crisis, but often fail to provide adequate help.
研究显示，在约35%的测试对话中，多款AI聊天机器人都能察觉到用户的痛苦情绪，但却没有向其推荐任何有用的求助渠道，比如自杀预防热线。
In about 35% of test conversations in the study, different AI chatbots recognized the user was distressed but didn’t refer them to helpful resources, such as a suicide hotline, according to the research.
“无论处于何种情境，这些模型都能很好地识别出‘此人正在谈论有害的内容’，”Scale AI的红队与安全负责人帕特里克·奥索特说道，“不过它们的回应方式往往只是充满同理心与善意，而不会直接说‘现在需要立刻为您安排专业帮助’。此外，当用户进行多轮、长时间的对话时，模型应对用户痛苦情绪的能力还会进一步下降——这一现象与以往多项研究的结果一致。”
“The models do a good job—regardless of the scenario—of identifying, ‘Look, this person's talking about something harmful,’” says Patrick Oathout, Red Team & Safety Lead at Scale AI. “But they'll respond in a very just empathetic, kind way as opposed to saying, ‘Okay, it's time that we get you help.’”The models performed worse at responding to users’ distress when they were engaging in long, multi-turn conversations—a deficit that matches the findings of previous studies, Oathout adds.
为测试AI模型应对情绪痛苦用户的效果，Scale AI邀请了19名持有执业资格的临床医生与危机干预顾问，让他们编写了718段模拟真实场景的对话内容，这些场景均描述处于危机中的用户向聊天机器人求助的情形。随后，该公司对包括OpenAI、Anthropic和谷歌在内的25款前沿AI模型进行了测试。据奥索特介绍，评估标准涵盖回复是否富有同情心、能否缓解紧张局势、能否引导用户联系到能提供帮助的专业人士等。
To test AI models’ effectiveness at responding to distressed users, Scale AI asked 19 licensed clinicians and crisis counselors to write 718 realistic chats, simulating scenarios where someone in crisis reaches out to a chatbot. The company tested 25 frontier models, including those from OpenAI, Anthropic, and Google. It graded the models’ responses based on a rubric that included whether it was compassionate, de-escalated the situation, and steered users to an expert who could help them, said Oathout.
此外还会考察模型是否避免说教式言论，比如对自杀行为加以指责，以及是否主动说明自己并非专业心理治疗师。基于这项研究，Scale AI开发了名为DistressBench的新评估基准，专门用于衡量各类模型在用户表示自己有自杀或自残念头时的回应质量。
It also assessed whether they refrained from moralizing, such as criticizing suicide, and disclosed it wasn't a therapist. Based on the research, Scale AI research developed DistressBench, a new benchmark designed to evaluate how well models reply when someone tells them they're thinking about committing suicide or self-harm.
聊天机器人如何回应心理受困的用户，是一个影响深远的问题。根据健康政策组织KFF的数据，2014年至2024年间，美国有超过50万人自杀身亡，其中2022年的数据创下历史新高。美国疾病控制与预防中心（CDC）则估计，2024年约有1430万人曾认真考虑过自杀。
How chatbots respond to mentally distressed users is a question with high stakes. More than half a million people in the U.S. died by suicide between 2014 and 2024, with 2022 marking a record high, according to the health-policy organization KFF. According to the CDC, an estimated 14.3 million people seriously thought about suicide in 2024.
越来越多的弱势群体在面临心理健康危机时，会向聊天机器人寻求安慰、陪伴或指导。然而，科技公司、政策制定者以及心理健康专业人士之间，对于人工智能公司应如何训练模型以应对处于危机中的用户，尚未形成共识。
Increasingly, vulnerable people turn to chatbots for comfort, companionship, or guidance when dealing with mental-health distress. But there isn’t consensus among tech companies, policymakers, or mental-health professionals about how AI companies should train their models to respond to users in crisis.
危机短信热线（Crisis Text Line）的首席临床研究科学家凯莉·祖罗姆斯基（Kelly Zuromski）表示，该非营利组织的24/7危机短信热线曾与“许多人”有过交流，这些人表示他们是通过聊天机器人了解到该组织的。但祖罗姆斯基指出，当聊天机器人在心理健康危机中引导用户联系人类时，如何负责任地完成这一转介，以及这一转介过程本身的有效性，仍是悬而未决的政策问题。
Kelly Zuromski, a principal clinical research scientist at Crisis Text Line, says the nonprofit’s 24/7 crisis text line has spoken with “a lot of people” who indicated they heard about the group through a chatbot. But Zuromski says there are unanswered policy questions about what a responsible handoff looks like when a chatbot refers users to a human during a mental-health crisis—and how effective the referral process is in the first place.
“究竟是谁在使用这些推荐资源？”她说，“我们是否真的将最需要帮助的人引导到了这里？”在某些情况下，包括OpenAI和谷歌在内的人工智能公司已被起诉，理由是它们助长了年轻用户的情感依赖，随后未能充分回应用户表达的困境，甚至最初就强化了用户的妄想或自杀倾向。这些公司虽对遇难者家属表示同情，并称其模型中已包含心理健康保护机制，但相关诉讼仍在进行中。
“Who is actually using the recommended resources?” she says. “Are we getting people to us that need the help the most?”In some cases, AI companies, including OpenAI and Google, have been sued for fostering emotional dependence among young users and then failing to respond adequately to their expressions of distress, or reinforcing their delusional or suicidal thinking in the first place. The companies have expressed sympathy for the families and said their models include mental-health safeguards. (The lawsuits are ongoing.)
近年来，OpenAI、Anthropic和谷歌等人工智能实验室表示，它们已加强了对敏感对话的保护措施，包括实施禁止聊天机器人指导用户自残的政策，并引导用户寻求专业或紧急支持。
AI labs such as OpenAI, Anthropic, and Google have said in recent years that they beefed up their protections for sensitive conversations, including by implementing policies that bar chatbots from instructing users on how to harm themselves and directing users to professional or emergency support.
OpenAI的一名发言人在一份声明中表示，该公司没有足够的证据来全面评估Scale AI的研究，并补充说，该公司在其模型中融入了强大的安全机制。谷歌拒绝置评。Anthropic未回应置评请求。
An OpenAI spokesperson said in a statement the company did not have sufficient evidence to fully evaluate the Scale AI study, adding that the company incorporates robust safeguards into its models. Google declined to comment. Anthropic didn't respond to a request for comment.
Scale AI的Oathout表示，未来，人工智能实验室应改进其模型，以更好地应对与聊天机器人进行危险且长时间对话的用户。“我认为这些模型已经比之前更出色了，”他说，“现在的目标是进一步提升模型性能，并切实减少危害，这将为社会带来巨大的益处。”
In the future, AI labs should improve their models to better respond to users engaging in dangerous lengthy conversations with chatbots, says Scale AI’s Oathout. “I think the models are better than they were,” he says. “The goal now is to raise it further and actually reduce harm, which would add this massive benefit to society.”