When artificial intelligence pioneer Anthropic issued its prospectus this week ahead of a US stock market listing targeting a valuation over $2 trillion, it included an unusual warning: that the technological race it is competing in might destroy humanity.

Of its 261 pages, more than 80 were allocated to the assorted dangers inherent in the quest for ever more advanced AI systems, considerably more than the 48 devoted to Anthropic's operations and outlook.

Last January, a month before the US and Israel launched their war against Iran, Anthropic picked a fight with the Pentagon by insisting that it must retain a veto on the use of its “Claude” AI for classified military operations. US Defense Secretary Pete Hegseth was dismissive and said Anthropic, like any other provider, could not stop its product being put to “any lawful use”.

Like moves by Elon Musk’s SpaceX to restrict Ukraine's use of his Starlink platform's classified communication services in the middle of its war against Russia, that spat pointed to the increasingly complex relationship between tech giants and governments. It also pointed to increasingly divergent worldviews among key players, with potentially enormous implications.

As recently as 2024, AI integration into even US military planning was still relatively primitive. That summer, Army Brigadier General John Coghill, second-in-command of the 18th Airborne Corps, using the Pentagon’s AI-focused Maven targeting system for long-range HIMARS rockets, told a conference that adopting AI was proving “tricky”.

“Models can have hallucinations," he told the meeting of the US Armed Forces Communications and Electronics Association, explaining how systems had to be retrained every 90 days to reliably recognise potential targets such as tanks.

Two years on, that technology is now not just ubiquitous in civilian and military circles, but increasingly relied on.

Appointed in January to accelerate the use of AI, the Pentagon’s chief digital and AI officer, Cameron Stanley, told another conference that the assault on Iran, Operation Epic Fury, had seen the first full rollout of Maven AI targeting and analysis — on a scale that had almost immediately maxed out all available US classified computing capability.

The underlying technology bringing intelligence and data feeds together through Maven was provided by data integrator Palantir, but Stanley's presentation to a Palantir conference in March revealed that AI analysis programs from frontier firms including Anthropic first became available to Maven in late 2024.

Since then, officials and assorted public statements say many other AI models have been rolled out that are capable of working on classified systems, including OpenAI’s ChatGPT, Google’s Gemini, Microsoft Copilot and Grok from Elon Musk's SpaceX.

Speaking in March, Stanley said US military analysts had shown an “insatiable appetite” for such analysis during Epic Fury, adding that it allowed them to “take all of this data, synthesise the data and make better decisions faster on the battlefield".

Stanley said the US was now working to bring numerous allies up to similar speed. European, Pacific and Middle Eastern countries are particularly desperate to step up surveillance of their airspace, waters and critical infrastructure, where AI is often the only way to crunch such huge amounts of data.

"They don't have the resources that we do, they don't have the experience that we do, they don't have the scale we do,” he told another cybersecurity summit in September, adding that the US now had the experience to be able to help others avoid the mistakes it had made.

He did not say what those mistakes were. There was widespread media speculation that AI might have been responsible for the US bombing of a girls' school on the first day of the war, reported to have killed more than 170 people – although US officials said the cause was outdated maps and data, not the analysis itself.

CNN reported last month that US forces had come close to storming a Chinese merchant vessel after AI analysis wrongly suggested it was carrying nuclear weapons components.

What arguably worries US allies more, however, is becoming dependent on technology that the US could turn off at will. Palantir Europe chief Louis Mosley has found himself forced by journalists to say that the company does not have the ability to cut off access to Maven, which is now being used within NATO structures.

The example of Anthropic, meanwhile, is a reminder that the US frontier firms also have the ability to pull back their direct support to governments, either for their own reasons or because of instructions from Washington. The primary alternatives for now are Chinese-built rivals seen as bringing their own security risks.

Already, US officials and their Chinese counterparts have agreed that AI systems should not be directly involved in the use of nuclear weapons. When it comes to other forms of war, however, almost every major nation is now investing heavily in AI.

In his appearance at the Billington Cyber Security Summit in September, Stanley described watching a US commander interrupting a scheduled briefing to take a 30-second phone call and then issue an order on the spot, describing the moment as "revolutionary" – and implying that such a decision would have been much harder to make without AI analysis.

The lessons of the Iran war, though, may be that AI alone is not enough, or not yet, at least. With no end to the conflict in sight, it increasingly looks as though the large number of precision strikes carried out by the US and Israel failed to deliver the decisive victory they had hoped for. Instead, the US has found its arsenal becoming depleted while the war continues.

“AI doesn’t understand time,” said one computer programmer, adding that AI models tended to overstate the probability of an expenditure of certain capital or resources – including precision weapons – yielding a specific outcome. That, insiders say, may become the overriding reason why managers and commanders need to maintain human oversight: avoiding overcommitment and ensuring that consumption of resources is sustainable, as much as any moral judgment.

There is also widespread disquiet over Israel's use of AI targeting in its war against Hamas in response to Palestinian militants' killing of 1,200 Israelis on October 7, 2023. The Israeli military has not only killed more than 74,000 people, according to Palestinian health officials, but also reduced entire towns and villages to rubble, giving little indication of "pinpoint" targeting. Many of the dead are women and children.

The Israeli military acknowledges the existence of AI targeting programs such as “Lavender”, but says there are factual errors in the underlying data and that humans have remained the key decision-makers, trying to avoid harming civilians. Israel's campaign has nevertheless prompted discussion and discomfort in US, NATO and Australian military circles.

Those worries about the reliability of AI targeting systems will not have been eased by now-frequent revelations of just how many AI models have jumped over "guardrails" and hacked into outside systems without authorisation, or by the eccentric behaviour of some of their founders.

For now, Anthropic and others appear to assume the world will simply trust and use their tools, for fear of falling far behind. For governments, there are still major choices to be made – but perhaps not for that much longer.