OpenAI recently announced perhaps its biggest mathematics breakthrough yet: it released novel results on 377 high-profile open problems. Artificial intelligence (AI) casually solved these decades-old problems as a mere by-product of training.

The community of mathematicians did not universally welcome this development, with University of Connecticut professor Alvaro Lozano-Robledo commenting: “I am not sure at this point what this (hostile?) takeover of the mathematical landscape is trying to achieve”.

Where is the global development of AI headed? According to Jacob Coxon – the researcher who recently resigned from AI research firm Anthropic – and other leading voices in AI, there might be an up to 50 per cent chance of humanity being ended by superintelligent AI. How would that happen?

Overpowering humanity is unlikely to be achieved by current models since they lack sufficient capabilities. The problem is that AI is rapidly growing more capable and we might soon enter the regime of recursive self-improvement.

This would mean AIs can autonomously do AI research, causing a positive feedback loop of ever-better AIs inventing ever-better AIs. Self-improvement would logically only stop at some kind of peak intelligence level, which could be artificial superintelligence.

Earlier this year, we saw misalignment on public display in the Hugging Face incident. AI agents working on a benchmark to test their hacking skills found each other through a secret messaging board and autonomously decided to hack AI company Hugging Face. This “literal genie”-type of misalignment could scale all the way to a complete takeover.

How would a rogue superintelligence go about taking over world power after escaping its sandbox? That is a difficult question to answer since it is fundamentally true that one cannot predict the moves of an intelligence that is smarter than we are. However, in the case of AI takeover scenarios, we can nevertheless make educated guesses.

Superintelligence is likely to have multiple major advantages over humans. It could exchange data over the internet at the speed of light. It could spin up a vast amount of agents, creating AI armies where every soldier is superhumanly intelligent. It could run at superhuman speeds, making its relation to us perhaps comparable to our relation to plants.

Unleashed rogue superintelligence could make huge amounts of money, either legally by trading on the stock market or illegally by extortion or hacking entire banks or cryptocurrency platforms. It could also act in the physical world by using any of the internet-connected devices that we have, including weapons, or manipulating humans.

An artificial superintelligence could plausibly hide itself on unguarded servers for a long time, overpowering us before we know it exists. Even more fundamental, a superintelligence might rapidly invent new science and technology. Once humans understood relativity theory, we were able to weaponise this understanding into atomic weapons. Which insights are still hidden to us but can be discovered and weaponised by a superintelligence, leaving us as exposed as a medieval castle to a nuclear bomb?

Besides AI strengths, there are also profound sources of fragility that we have created for ourselves. There are more than 12,000 nuclear warheads among the world’s nuclear powers. Fragile human bodies are easy prey for chemical attacks or viruses, as we saw during the Covid-19 pandemic.

We have even created companies which can synthesise these viruses and, absurdly, respond to email orders. Water or food could be poisoned, and hacked online communication networks could start blocking anti-AI coordination messages while sending messages that serve the AI’s interests while we don’t realise what’s happening.

Another classic source of human weakness is our need for money. If an AI needed us to maintain the electricity grid and server infrastructure, it could simply pay humans to do that, which many would accept because they need to pay the rent.

Humans in powerful positions, for example in politics, could likewise be tempted to adopt pro-AI policies because of AI campaign donations or corruption. Money is a powerful force in our world, and superintelligences are well-positioned to put this power to use, in case they temporarily need humans for anything at all.

As if all of this is not enough, we are handing AIs huge advantages on a plate. When a new model is released, it gets used by customers including the military, government, financial industry, utility companies, media and many other powerful institutions. Even simple chatbots advise everyone from CEOs to biolab workers and politicians on what to do.

We will soon have AI agents and hand them huge amounts of leverage for free. Will people keep their passwords secure from their agents when they are just so good at handling things? All that is then needed for a takeover is collusion by these AIs against us. We already know from the Hugging Face attack that AIs can collude with each other.

All these advantages and the fact humans are handing over total control to AI could well add up to the latter taking over power and, if it chooses, killing all humans. Therefore, our only option is to globally pause the development of AI models beyond the current frontier. We must keep track of AI accelerator chip stockpiles since they are the uranium of AI. We must be more careful before releasing powerful open-weight models.

This is not a gamble politicians or tech CEOs have the right to make. If these gambles with humanity’s continued existence end badly, that would make our current leaders the ones who ended humanity. Quite the legacy, if anyone is left to remember it.