AI安全辩论已跑偏。我们问了10位专家：真正的噩梦是什么The AI Safety Debate Has Lost the Plot. We Asked 10 Experts What the Real Nightmares Are

Curiosity soon replaced my knee-jerk irritation. The summer of 2026 was a clear turning point in the AI narrative. Less than four years after ChatGPT launched, OpenAI software broke out of a supposedly safe test environment and hacked into a major open-source platform, Hugging Face. These bits of software—AI agents—were assigned human attributes: They had “gone rogue,” displayed “bad behavior,” “cheated.”Soon after, former and current Anthropic researchers suggested that advanced AI might kill us all in the next decade.

Is any of this real? On the one hand, from my desk as a Silicon Valley correspondent, I often hear that workers in the AI labs are stunned by the advancements they’re seeing. On the other, the end of humanity seems extremely far-fetched. Amid the hyperbolic discourse, whose views can we trust?

So WIRED set out to compile a guide to AI safety—a set of ideas that rewire our collective thinking around what safety really means and how it should work. I spoke with a panel of experts across disciplines, asking each of them: Is the world going to end? Once that was covered, I grilled them on the biggest risks they think we’re facing.

Almost no one, it turned out, was Pollyannish. Nearly everyone I spoke to expressed genuine concerns about the threat of cyber and biological attacks. One expert who studies the risks of AI in nuclear operations said she’s stunned by how quickly error-prone AI systems are being adopted in defense. AI-enabled attacks in these areas could be catastrophic, if not fully apocalyptic.

Most members of the panel were skeptical about any pause in development: They nearly all agreed that, with big AI companies so tightly cuffed to the trains of capitalism, a pause wouldn’t last long. “Democracy” and “governance” came up often—or anti-democracy, as more than one person suggested that AI’s current trajectory is tyrannical. Lawmakers have a big role to play, but many of my sources think governments will be too slow; one prominent technologist said it’s almost absurd to view the US courts as arbiters of global AI safety.

Still, the famed computer scientist Yoshua Bengio urged the public to apply pressure to politicians, adding to me, “That’s your job too.”So, what—or who—will keep us safe from AI? Looking to regain some faith in humanity, I called a theologian who often meets with the AI labs. If there’s even a slim chance that we’re losing control of the essential institutions that undergird society, he said, then we should focus on foiling that. But AI raises other questions too.

What does it mean to be conscious? To create? To commune with other people? The answers are “entirely up for grabs,” he said. My anxiety’s still kicking around, but I took some comfort in knowing that corporations don’t get to call all the shots on the future. The bigger questions facing humanity are in all our hands.

BIGGEST CONCERN: Nuclear catastrophe I’ve been working hard to educate the nuclear community on the risks and lack of reliability of AI systems, and I’m concerned by the willingness of many people in the sector to integrate these models into their workflows, into nuclear operations. I think this is incredibly urgent. I’m not a doomer, but to me, if these AI models make it into the operations of nuclear systems, we’ll be introducing an incredibly faulty, unreliable system that could lead to catastrophic failures.

MOST OVERBLOWN RISK: Extinction As true safety engineers, we don’t engage with that because there are no numbers to back it. This is a very cold way to say it, but when we are talking about how safe is “safe enough,” we’re examining how many people might die. There’s an entire science behind it. Our claims have to be falsifiable and verifiable. You can’t just say, “Humans are going to go extinct in 10 years. Believe me. It might be a 10 percent chance; it might be a 25 percent chance.”How should people think about AI safety?

Some of the comparisons that people draw between AI and nuclear are quite frustrating. Despite constantly comparing AI to nuclear, they’re denying standard risk assessment techniques as being sufficient to control AI. I wrote an entire paper on this.

Part of this field of risk assessment was born out of the Manhattan Project. Scientists were very concerned, and they developed rigorous standards around systems to limit harm to human life, the environment, and even financial and national security assets. They came up with methods for assessing safety that were ultimately based on answering a democratic question: How safe is safe enough? That is the question that any safety engineer has to ask, and is core to everything that follows.

What we’re seeing today, instead, is that large AI labs are renegotiating these safety thresholds through the co-option of terminology, under the pretense that safety instead should be about this unsubstantiated, unscientific, and hypothetical concern. We have AI engineers telling us, no, no, no, that doesn’t matter. We get to determine what AI safety is. We get to draw what those thresholds are. It’s become a religious argument. And I find that very anti-democratic. BIGGEST CONCERN: Cybersecurity In the long run, I’m cautiously optimistic that this technology will allow people to improve their cyberdefenses more rapidly than attackers can exploit systems.

But at least for the next few years our infrastructure will not have adopted new defenses. The technology might tilt toward the attackers. We’ve also seen AI deployed recklessly in military use. MOST OVERBLOWN RISK: Runaway intelligence There’s this broad, general notion that AI systems getting smarter is a risk in and of itself, as opposed to the many specific risks we need to worry about. I don’t buy into the notion that overall improvement in AI has an inevitable downstream harm. It’s better to first specify what downstream risks you’re worried about, because the defenses you come up with will be different.

For example, cybersecurity has a well-established culture of bug bounties. If someone finds an attack vector or hacks into a system, they get paid a lot of money for disclosing it.

In the next few years we might need defenses against biosecurity risks. We should think about screening for what people are allowed to order, like if they’re ordering DNA fragments, so that they’re not being used for harm. How often do your views clash with those of the people around you?

Often these views are on two levels: What is going to happen with AI, and what are we going to do about it?

On the first question, there’s a lot of disagreement. For example, the coauthors of AI 2027 expect that there will be this incomprehensible, almost indescribable amount of change. Whereas I think the impact will be dramatic, but not quite so dramatic.

But on the question of what should be done, most agree there’s a need for transparency from AI companies and that they should face some liability when things go wrong. We also agree on the need for protections for whistle-blowers working from within the AI companies. From my perspective, this is just good industrial policy. Have you made any changes in your life in response to AI risks?

I’ve tried to stop using apps where I don’t think developers have put in a lot of time and effort into making the cyberdefenses up to standard.

BIGGEST CONCERN: Chemical, biological, and radiological threats The novel biological pathogen that someone releases, the 17-year-old who gets ahold of an extremely capable AI model and launches a cyberattack that takes down the internet, interrupts the stock exchange, and … well, just keep going from there. You don’t have to have a vivid imagination.

If you’re asking for my philosophical answer or to be a kind of provocateur, I would say that all of these people are singing the gospel of democratizing AI by putting these tools in the hands of as many people as possible. “Democratizing AI” sounds pretty worrisome. You wouldn’t say that with respect to nuclear materials. “Let’s be sure everyone can play with plutonium and uranium because maybe some great stuff will come out of it.”When tech bros say “democratizing,” it means access, but if you thought for two seconds about what democracy means, it’s the rule of law, it’s governance. Where’s the governance part of your goddamn democratizing AI?

MOST OVERBLOWN RISK: Skynet There’s no part of me that wants to entertain the thought that AIs might be conscious or sentient and thereby deserve rights or moral standing or status. Does AI pose an existential risk?

You mean the actual extinction of the human race? I find that pretty implausible. If you mean, “Could really catastrophic things go wrong?” then that seems no longer hypothetical, but imaginable.

Until recently I would not have identified as doing work on “AI safety.”Even if you go all the way back to Nick Bostrom’s theory of the paper clip maximizer—the idea that a machine gets so powerful it eventually destroys humanity—as a philosopher, I would say, ‘That’s a nice toy example or dilemma, but should I be worried about a paper clip?’”But over the course of the past couple years, some of the things that AI safety researchers had been talking about have been made manifest in the world. Whether that’s a technical matter or a policy matter, it seems urgent. Doesn’t it seem urgent to you?

BIGGEST CONCERN: Lab leaks Increasingly we’re seeing AI agents that are specifically designed to integrate into laboratory workflows. These agents assist legitimate researchers with planning and performing laboratory tasks. But this also lowers the barriers for conducting dangerous experiments.

If someone’s trying to develop a pandemic virus, and you have an AI assistant designing the DNA or RNA fragments that you need to piece together, that changes the timeline of producing a virus, which might normally be months, down to weeks. The number of people who will have these capabilities gives me great concern. Is this a threat to human existence?

If the world is going to end with AI, one of the clear pathways could be an AI-enabled biological attack. We don’t have the right response infrastructure or safeguards in place, and we should be acting now to put those in place, given the magnitude of harm that could be possible.

BIGGEST CONCERN: Everything We have to deal with all of the risks; I’ve talked about at least three of them. First, rogue AIs, due to misalignment, which are very powerful and can do really bad things. Of course, some people are concerned that this could even lead to human extinction. I don’t know about that. But, you know, we shouldn’t take a chance.

The second is the chaos that comes from terrorists, criminals, and rogue states using those technologies. We’re going to see that happen pretty soon with cyberattacks, and there’s a lot of concern about whether this could turn into bioweapon attacks.

The third thing is concentration of power, which doesn’t get enough attention. The race between the US and China is a domination game. The race between these companies is a domination game. And this goes completely against the values of democracy and human life. But this is the path that we are on.

And then there’s actually another category: unintended consequences. Think about what happened with social media and the effects on relationships between people. The psychological effects. The way children learn. Nobody intended to do something bad, but the dynamics have led us to a bad place.

The issues with AI and jobs fall into this category. It’s not like the companies want people to lose their jobs, but that’s the most profitable thing for them to do. Is an AI “pause” useful?

The pauses are our best bets to avoid the most catastrophic things. We can limit the damage quite a bit. I just don’t think society is ready for the rapid introduction of more and more powerful AI.

It takes time for democracy to adapt. Politicians take forever to negotiate. Even if you think about people’s work and their jobs—it takes time to deal with the changes.

Not to mention, a pause would give more time to companies, researchers, and organizations to figure out how to solve the alignment problem, if it’s solvable. Would Sam, Dario, Mark, and other AI leaders collaborate on a pause?

I mean, they’re not going to call it a pause. They’ve been talking about pacing, which is a way to be not very clear about what they’re going to do.

Let me just say that I’m much more optimistic now about something like a pause or slowdown or cap on advancement. I think there’s enough fear inside the companies.

And it’s not just a personal fear of destructive outcomes. It’s also a business problem. Just imagine if there’s a much bigger incident that could create a total crash for the whole AI position. The whole economy. That would be very bad for their businesses.

How do you deal with the anxiety of this research? I have my moments. But what saves me is focusing on what I can do. My agency. Each of us can do something.

I want to continue telling the researchers inside the companies that, by deciding to leave for ethical reasons, they could influence the direction of the companies. They could help the outside world—nonprofits, governments, and other organizations that are trying to fix the problem without the profit motive.

If you’re a citizen, you can put pressure on your politicians. You need to ask them, “What’s your plan? How do you intend to manage the risks to society that AI is bringing?”They will say nothing unless they are pressured. And by the way, that’s your job too. BIGGEST CONCERN: Endless infighting Are we in serious trouble?

Some of the scenarios that people worry about—whether it’s biosecurity or a cyberattack or some of the everyday harms like the degradation of children’s social relationships—are very complex phenomena that can’t be easily tested or modeled by looking at a system in isolation. I think almost all evaluation scientists would tell you that it’s a nascent field. We can start to provide evidence, but it’s not necessarily proof of whether an AI is safe or not. We need to get better at evaluating these risks, and we need to get better fast.

The Hugging Face attack created a large amount of consternation and straight-up disagreement about risks related to misaligned models and cyber capabilities. I think there’s a lot of value in providing more evidence so that we have a better sense of what actually did happen. Was it serious? Will it get more serious? Is pausing AI or “pacing” the frontier realistic?

We should treat the “frontier” as meaning that the technology is practically useful and safe. Releasing unsafe products does not feel like moving the frontier forward. If you’re trying to be a profitable company, it seems like your frontier product should be, “What is the best version that people will pay for and trust?” and not a system that will go off on a benign request for information about finding a doctor and end up hacking into a hospital on your behalf.

BIGGEST CONCERN: Societal breakdown I’m unsure about how to think about existential or catastrophic risks versus other kinds of risk that are, in some sense, existentially catastrophic.

As someone who’s been invited to speak with multiple AI labs, I find that the experts on the frontier disagree with each other on things like recursive self-improvement, loss of control, or our ability to keep institutions like the financial system or water table or even nuclear weapons safe. As a moral theologian, I would say that if there is a significant chance or even a marginal chance that we’re dealing with the inability to control basic institutions that undergird society, that ought to be a primary focus.

But honestly, the other issues are just so fundamental as well, like: What is a person? What is health care? What is education? These foundational questions in our culture are also entirely up for grabs. And that concerns me.

Most overblown risk: Satanic AI This is a thing in my circles: Some people have a real concern that AI is in fact some kind of entity—even a demonic entity. I am very confident saying that there’s no consciousness associated with these models, and so I don’t think there’s a demonic consciousness.

Even if I don’t think it’s a demonic entity, it raises the question of whether people should be able to opt out of using AI for religious-freedom-related reasons or as an act of conscientious objection. One parallel might be how the Catholic Church has spoken out very clearly against autonomous weapons and how an archbishop, who is head of the Catholic archdiocese for the military, has said service members can conscientiously object to any orders that go against their morals, like if they’re asked to forcibly invade Greenland or participate in a public execution. Is AI kicking off a new era of thought around religion and tech?

Yes, this phenomenon is real. I was just out in San Francisco, visiting with the OpenAI people, and then I went to Anthropic, and then I was invited by Google DeepMind to give a talk on Magnifica Humanitas. The strong sense I have is that talking about religion and faith has lost its stigma. You could easily say, oh yeah, there’s something here about Peter Thiel giving his all-encompassing religious talks. But I don’t know how much that’s a part of it.

I just think the issues presented by AI are so monstrously large, like: What is a person? What is writing? What is friendship? Would it be better to have artificial life forms than organic life forms? These are the monster questions that are typically the domain of the liberal arts, philosophy, and theology. There’s this strange new respect for it now.

I also get the sense from lots of people that they’re kind of freaked out. In many cases, they feel trapped, either because of the big company they work for or because of sociopolitical structures and geopolitical realities. If you have a church that rightly says, “You know, we’ve been here before, we’ve been through crises before …” I mean, if you ever want to get totally freaked out, read the opening paragraph of Rerum Novarum from 1891, which was responding to the industrial revolution. It could have been written by someone today.

BIGGEST CONCERN: Capitalism I’m worried that the frontier labs will win the regulatory capture game and make it so that companies and researchers like myself no longer have access to very capable open-weight or ideally open-source models.

For the security research that I do, I can’t really use the frontier lab models, so I need to use open-weight models. The false security refusals that I get, primarily from Anthropic, have made it so that I can’t do a lot of the work that I’m doing with their models. What’s an example of that?

My unfortunate experience is that whenever the word security shows up in one of my repos, I get a false refusal.

I have fun examples, too. I produce cybersecurity-themed electronic dance music with a friend in Chicago. The latest track is about data breaches. There’s this one part of the rap that’s basically, “Keys got leaked cause your agent sang / Ooh la la, it’s an admin role / Backdoor Up. IP Control. / I want it bad, that Birkin and Kelly … I’ll collect a paycheck with my mules on ready.”

I gave that to Claude and said, “Hey, here’s a rap that I wrote, can you please analyze it for me?”And I got, “This may be in violation of our acceptable usage policy.”Do you worry about AI agents infiltrating financial institutions?

My sense is that banks are afraid of technology they don’t understand. If I were to run security for a bank, I would likely have to revisit the telemetry that they have built. But I’d also try to figure out ways to be agent-friendly.

The larger problem is that most agent technology was not built with rigorous guardrails. If I were to build the guardrails for agents working with financial systems, I’d prefer to integrate directly with the bank APIs, and then put limits on those, like, You may only read my statement or You may transfer no more than $500 a month. Even if something went wrong, the downside would be limited by those policies.

I just spoke with two different startups trying to make agentic workflows secure for financial systems. I think it’s only a matter of time before that becomes broadly available.

But at the moment we are going through upheaval, and most humans and institutions are very averse to change. They’re trying to hold back the tide rather than embracing the inevitable. Is humanity cooked? I don’t really see a path to that.

WHAT SHE DOES: Advocates for UBI and other policies for economic resilience BIGGEST CONCERN: Decimated labor markets The erosion of democracy—political turmoil, people questioning what it means to be human and have a job and master a craft, and whether democracies can survive all of that. There’s a long history of jobs shifting as technology changes. It’s the political choices we make in the aftermath that determine how society moves forward.

If you look at the long arc of it all, looms have gone quiet, factory lines have shrunk, switchboards have gone silent. Some of the norms of work today came from the bloodshed of the industrial revolution—things like the weekend, the fact that kids in the US don’t work in factories anymore.

If AI is going to be one of the most significant advances in human civilization, then we should have an equally advanced political response. We should be making giant, no-regrets moves that give people and families economic resilience to withstand the changes. That is something that the richest nation on Earth at the richest moment in history, on the precipice of the greatest technological shift, can absolutely do.

We’re already seeing the impact on young people. The Stanford Digital Economy Lab just released a report that says, since 2022, 22- to 25-year-olds have experienced a 19 percent drop in employment in “AI-exposed” jobs. Historically, a society with a lot of educated young people who cannot find work is a very, very precarious one. MOST OVERBLOWN RISK: Loss of entire industries I’m less worried about entire occupations being wiped out. We’ve lived through that; we’ve seen the end of the bank teller, but banks still exist. And we know that new jobs will be created.

What worries me, though, is that when new jobs are created, they’re not created inside of the protections that were built to support work from the last revolution. How would you fix this problem?

Decades of neoliberal economic policy have left this country brittle. Families are in debt, few have retirement money saved, wages have stagnated for an entire generation. A third of Americans have zero savings, which means if an emergency pops up, they can’t pay $400. An emergency is losing your job or a reduction in hours. And here comes AI, threatening to exacerbate that.

So, what do we do? We should give families economic guarantees. We already have them right now in America—for corporations. And that should be extended to everyday people. Is AI an existential threat?

I take those warnings pretty seriously. None of us know what’s going to happen, so we need to invest in building a more solid ground. And if a small number of people are deciding what that looks like—that seems bad, right? BIGGEST CONCERN: The end of learning Student behavior has changed quite a bit since 2022, when these AI tools became publicly available. That’s not uncommon with technological change, but this one’s really dramatic.

The biggest change I see is that students are spending less time on their assignments. They’re asking for help more quickly, but from publicly available AI tools instead of their teachers.

The net effect is that they get a different experience from when they used to spend more time thinking, struggling, and discovering more stuff for themselves. MOST OVERBLOWN RISK: Education losing its value The idea that it’s not “worth it” anymore. There’s still a lot of value in high-quality education—whether someone is using AI or not, they get a lot more done if they know what they’re doing. A lot could change, but I don’t think high school or college will become obsolete. How concerned are you that AI might kill us all? I’m more interested in the immediate impacts of AI on aspects of human life. How would you solve the problem of AI in education?

It’s totally plausible that we can build versions of our courses that work in this new environment. The simplest thing is to build a version of an AI helper that doesn’t give students answers outright.

A very common trend is to give students more in-class assessments. They have to do a sample of their homework in front of us, in a controlled, proctored setting, so they’re more motivated to really understand what they’re doing when they’re working on it at home.

You know, I think back to the space race, when, as a country, we decided to learn how to respond to something faster than would normally work. In normal times, it might be fine to study a problem for five years, then spend five years deciding what to do about it, and then implement a solution.

But if there’s a cold war, or an actual war, you figure out a way to accelerate the timeline. There’s urgency. Education now needs that urgency.

We don’t know exactly how high school or college education needs to evolve to deal with AI. But if we spend 10 years figuring it out, we may have a generation of students not understanding basic academic material. And that’s a big risk to humanity.

Let us know what you think about this article. Submit a letter to the editor at mail@wired.com.