"I have this sense that, within a few months, AI is going to be all anyone is talking about," said one cabinet minister, with an ominous stare.

It was Labour conference, with everyone from ministers to activists jubilant at Andy Burnham's takeover and a bounce in the polls after the change of leader.

But a number of senior Labour figures were privately starting to name the power of AI as the thing worrying them most, alongside the immediate pressures on the cost of living and fragile economy.

One cabinet minister – with a brief unrelated to technology – told the Guardian that dangerous, out-of-control AI was the issue keeping them up at night, fretting about the threat to humanity. It was "not killer robots" causing the worry, they explained, but the idea of a huge amount of concentrated, super-intelligent power potentially ending up in the hands of bad actors.

Another expressed concern that the government was "falling behind the public" in terms of dealing with worries about AI safety – and that it needed to catch up fast.

In his conference speech, Ed Miliband was the most strident about the "daunting" dangers of AI, making the case that politicians could not and would not allow the profits of tech companies "to come at the expense of our children or indeed humanity".

And it has not taken long for these alarm bells to ring even louder, with the last seven days looking like the week Westminster fully woke up to the warnings.

A flurry of activity started on Monday with Kanishka Narayan, the AI minister, stating it was "likely, if not inevitable" that rogue agents would hack the British state, and saying the UK was looking to strengthen its forthcoming cybersecurity legislation.

It was the main topic of discussion at cabinet on Tuesday, and that was followed by the MI5 chief Ken McCallum warning AI was "supercharging the cybersecurity contest between attackers and defenders". He said the biggest threat was extremist groups using the technology to unleash bioweapons – toxins used to cause death and disease.

Liam Byrne, the chair of the business, innovation, science and trade committee, said the change of government from Keir Starmer to Burnham was partly behind the increased focus on protecting the UK from AI threats but more still had to be done. "The reality is that the Starmer administration was much too close to the big tech companies," Byrne said. "And Burnham is not, frankly, so there has been a change of posture. Those things, plus the security services issuing their warning, that is what has changed the weather on this."

One senior Labour government figure agreed that Burnham's government was taking AI safety warnings more seriously. "I do feel there has been a shift. Under the last lot, too many of them had drunk the tech-bro Kool-Aid," he said.

Byrne's committee held an evidence session this week with civil service leaders in charge of AI security and executives from the key companies –Anthropic OpenAI, DeepMind and Meta – with a split between them about whether government testing of new models should be mandatory.

He said his committee was "lighting up" the issue, particularly in terms of working with the US to regulate and monitor new models. It had been reported that Donald Trump's administration had asked AI companies not to share their latest models with the UK's cutting-edge AI Security Institute for testing, though Narayan confirmed this week that they had managed to retain cooperation.

But Byrne said the government now needed to seize the moment and take the lead on new regulation, particularly if Trump's Republicans lost control of Congress in the midterm elections to the Democrats.

"I think the government is still caught in a dilemma between how to keep model access, given a capricious US president, and the safeguards that the industry itself now says are essential. It's make-your-mind-up time, I would say," Byrne said.

"The companies want to cooperate … It's not in their commercial interest to issue models which are lethal. Now they too have got to navigate the White House. So I just think this is a moment when the government has actually got to do what is right for the country because Congress is about to change the dynamics on this in the US." The UK's upcoming G20 presidency will be the key moment to push for international action to rein in AI.

Geoffrey Irving, the former chief scientist at AISI who spoke at this week's parliamentary hearing believes Whitehall concerns are justified. OpenAI and Anthropic's technology has already shown its hacking capabilities but their models are released with safety guardrails in place that at least mitigate against the threat of malicious behaviour.

However, so-called open source AI models are freely available and can be adapted. Typically, they caught up with the paid-for models such as OpenAI's within less than a year, said Irving, although they could cause disruption already. "It's very likely that open source models are already powerful enough to do serious damage," Irving told the Guardian, adding that he expected open source technology to reach the capabilities of OpenAI and Anthropic products "soon". Safety guardrails "either don't exist or can be trivially removed" from open source models, he added.

The business committee heard this week that AI-enabled bioweapons could become the next serious threat to manifest itself, in the wake of cybersecurity, as models become more powerful and therefore capable of designing novel bio-harms – although experts suggest there is a gap between designing dangerous pathogens and weaponising them.

Concerns over cybersecurity and bioweapons can escalate rapidly to fears about AI wiping out humanity in a variety of scenarios – a concern that is widespread within the AI community, and is shared by Irving.

However, some experts argue that the government should focus on immediate risks from AI. Prof Elena Simperl, a co-director of the Institute for Artificial Intelligence at King's College London, said the existential risk argument had "no evidence" and was a "theory" rather than a credible threat.

"There are bigger, more immediate risks that AI technology is posing now and are actually evidenced, like hacking and signs that AI can be as addictive as social media," Simperl said.

In the absence of international AI regulation, there are some in Whitehall worried about whether the UK is doing enough preparation to defend against AI warfare and attacks.

Defence sources said there was concern about the cancellation last year of the AI programme at the government's Defence Science and Technology Laboratory (Dstl), based at Porton Down, with worries that this was shortsighted given the need for cutting-edge research on advancing the nation's capabilities and defending against its risks.

This was countered by an MoD spokesperson who said AI was "transforming defence from industry to the battlefield and will play a decisive role in how we fight and defend" and that it must be "integrated across all workstreams, rather than a standalone programme". They pointed to a £400m investment in AI-focused capabilities over the next four years to "prepare the UK for the future of warfare".

Others in Whitehall have the opposite concern: that worries about safety could become too dominant – and end up holding back the government's adoption of tools that could be transformative for productivity and solving longstanding problems in public services, from medical care to reducing waiting lists.

The former cabinet secretary Gus O'Donnell told the Guardian that ministers needed to be as aware of "the risk of being left behind" when it came to AI as being aware of the need for guardrails.

Launching a Global Government Forum report on AI with the tech firm Cognizant, O'Donnell told the Guardian: "There is a big need to get it better and to really push on. I'm really optimistic, but the challenge for us, obviously, is managing the risks along the way, making sure that we don't make mistakes, which reduce people's trust, which means that that sets us back."