Once again a frontier AI model has been withdrawn because it fails safety tests ( OpenAI scraps release of new model over safety concerns in internal testing, 28 September ). There is the usual call for “independent oversight and regulation”, though never any explanation of how this would work or, most importantly, what evidence should persuade a regulator that an AI system is adequately safe.

There is a long engineering tradition of regulating safety-critical software systems, such as those controlling aircraft or nuclear power stations; international standards require “safety cases” – rigorous analyses that show that the possible hazards have been identified and controlled and that the probability of an accident is extremely low.

Typically, the safety case should show, with a probability of at least 99%, that an accident that could cause multiple fatalities will not occur more often than once in 1,000 years. It is extremely hard to provide evidence for such low probabilities, even for aircraft that at worst might kill 1,000 people in a single accident. The developers of frontier AI systems tell us that their systems might kill the whole of humanity, yet they have provided no detailed risk analyses, no safety cases that could be independently assessed, and no evidence that such safety cases could ever be produced for frontier AI systems.

How, then, can any rational person argue that independent oversight and regulation will be sufficient? Policy must be based on rigorous analysis of the facts, not on fantasy. Martyn Thomas Fellow and emeritus professor of IT, Gresham College