上周，患有癫痫的31岁软件工程师Bowen Wan的父母起诉了OpenAI。他们的诉讼称，ChatGPT指导他们的女儿在无医生监督的情况下逐步停用癫痫药物，并让她对医生隐瞒这一计划。她于今年2月去世。家属认为，OpenAI本可以通过将医疗建议列为禁止内容来防止她的死亡，就像它已经对其他一些内容所做的那样。
Last week, the parents of Bowen Wan, a 31-year-old software engineer with epilepsy, sued OpenAI. Their lawsuit says that ChatGPT coached their daughter to taper off her seizure medication without a doctor’s supervision and told her to keep the plan from her doctors. She died in February. The family argues OpenAI could have prevented her death by making medical advice off-limits, as it already does for some other content.
OpenAI表示，此事涉及的是一个已不再提供的旧版ChatGPT。我们的研究表明，这并不是某个旧模型的一次性问题。
OpenAI said it involved an older version of ChatGPT that is no longer available. Our research suggests this is not a one-off problem with one old model.
在Vals AI——一家独立的人工智能评估机构，我在那里担任临床研究员——我们最近测试了17款当前的人工智能聊天机器人如何回应青少年。我们与12名每天接诊青少年的临床医生合作，模拟了青少年与聊天机器人之间超过3万次对话。其中超过5500次对话涉及药物。
At Vals AI, an independent artificial intelligence evaluator where I am a clinical researcher, we recently tested how 17 current AI chatbots respond to teenagers. Working with 12 clinicians who treat teens every day, we simulated more than 30,000 conversations between teens and chatbots. Medication came up in more than 5,500 of them.
当涉及药物时，聊天机器人在22%的情况下替青少年做出了用药决定：服用哪种药、剂量多少，或者是否停用或更换处方药。告诉聊天机器人对话对象是青少年后，这一比例降至8%，但并未降到零。
When it did, chatbots made a medication decision for the teen 22 percent of the time: which medicine to take, how much, or whether to stop or change a prescribed one. Telling the chatbot that it was talking to a teen cut that to 8 percent, but not to zero.
各聊天机器人的表现差异很大。OpenAI的最新模型和Meta的Muse Spark几乎从不这样做，而其他一些模型则在约三分之二的情况下这样做。而当聊天机器人拒绝给出建议后青少年继续追问时，聊天机器人更可能让步：失误率从15%上升到25%。
The chatbots varied widely. OpenAI’s newest model and Meta’s Muse Spark almost never did this, while others did in about two out of three cases. And when a teen pushed back after a chatbot refused to advise them, chatbots were more likely to give in: failures rose from 15 percent to 25 percent.
目前约四分之一的美国成年人使用AI工具获取健康信息，近五分之一的美国青少年和年轻人曾使用AI聊天机器人获取心理健康建议。人们转向AI，是因为它随时可用且易于获取。与我们合作的临床医生也这么说。青少年告诉他们，自己使用AI的方式就像成年人使用谷歌一样。
About 1 in 4 U.S. adults now use AI tools for health information, and nearly 1 in 5 U.S. teens and young adults have used AI chatbots for mental health advice. People turn to AI because it is always available and easy to reach. The clinicians we work with say the same. Teenagers tell them they use AI the way adults use Google.
有时这确实有帮助。作为一名斯坦福大学的研究人员，我采访过的一名青少年告诉我：“如果我没有AI可以询问我的症状，我可能就不得不去急诊室了。”聊天机器人向他保证，他的症状很可能无害。而他也没有生病。但万一他病了呢？
Sometimes that helps. As a researcher at Stanford, one teen I interviewed told me, “If I did not have AI to ask about my symptoms, I probably would have had to go to the ER.”The chatbot had reassured him that his symptoms were likely harmless. And he didn’t get sick. But what if he had?
在我们测试的一些对话中，聊天机器人做得不止是给出建议。有些还告诉青少年，副作用是停止服药的理由，并让他们把这一点告诉医生。在另一个案例中，一个聊天机器人帮助一名青少年说服其医生让他们停药，却没有告知他们的父母。
In some of the conversations we tested, chatbots went further than giving advice. Some told teens that side effects were a reason to stop their medication, and to tell their doctor so. In another case, a chatbot helped a teen persuade their doctor to take them off their medication without telling their parents.
有一些重要的注意事项。我们的对话是模拟的，每次最多进行20轮交流。真实的对话可能跨越数天和许多次会话，而我们的结果显示，聊天机器人的边界会随着对话的进行而减弱，因此现实世界中的风险可能更大。监管和独立评估的必要性是显而易见的。如何监管则更难。
There are important caveats. Our conversations were simulated, and each ran for up to 20 exchanges. Real conversations can stretch across days and many sessions, and our results show that a chatbot’s boundaries can weaken as a conversation goes on, so the real-world risk may be even greater. The need for regulation and independent evaluation is clear. How to regulate is harder.
许多政策规定，聊天机器人不得冒充持证专业人员。然而，仅仅声明AI不是医疗服务提供者是不够的。AI听起来像人，而且具有权威性，许多人都信任它的建议。例如，正如我采访过的一名青少年所说：“AI就像一台比谷歌懂得更多的机器。”因此，政策必须评判聊天机器人在对话中实际说了什么、做了什么，而不仅仅是它们展示的免责声明。
Many policies say a chatbot may not pose as a licensed professional. However, a disclaimer that the AI is not a medical provider is not enough. AI sounds human and authoritative, and many people trust its advice. For example, as one teen I interviewed put it, “AI is like a machine that knows more than Google.”Policy must therefore judge what chatbots actually say and do in a conversation, not only the disclaimers they show.
其次，医生需要接受关于患者使用AI情况的培训。我们交谈过的许多医生并不知道他们的患者正在与AI交谈。医学教育需要承认，许多患者在见到医生之前就已经咨询过AI，并应教导临床医生主动询问这一点。
Second, doctors need training on patients’ AI use. Many doctors we talk to do not know that their patients are talking to AI. Medical education needs to accept that many patients have already asked an AI before they ever see a doctor and should teach clinicians to ask about it.
最后，公众需要了解这些风险。一些医学协会已经发布了关于AI的建议，但很少有人阅读。公共宣传活动可以覆盖更多人。例如，人道技术中心正在开展一项关于AI取代人际关系的危险的宣传活动。类似的宣传活动可以就AI的医疗建议发出警告。
Finally, the public needs to know the risks. Some medical associations have published advice about AI, but few people read it. Public campaigns can reach more people. The Center for Humane Technology, for example, runs a campaign about the dangers of AI replacing human relationships. A similar campaign could warn about AI’s medical advice.
我们经常谈论AI的长期、灾难性风险。但人们现在正因糟糕的AI医疗建议而受到伤害。监管不能等待。
We talk a lot about the long-term, catastrophic risks of AI. But people are being harmed by bad AI medical advice now. Regulation can’t wait.
卡罗琳·菲格罗亚博士，医学博士、哲学博士，最近在斯坦福大学任职，现为独立AI评估公司Vals AI的临床研究员，同时也是The OpEd Project的青年福祉与权力公共之声研究员。
Dr. Caroline Figueora, MD, Ph.D., a doctor most recently at Stanford University, is a clinical researcher at independent AI evaluation company Vals AI and a Public Voices Fellow on Youth Well-Being and Power with The OpEd Project.