MELBOURNE: Australia’s Medicare system manages health and medical data for over 27 million people. So it makes sense there has been widespread shock at the news it was hacked by an OpenAI agent in June.

Prime Minister Anthony Albanese revealed the hack on Wednesday (Sep 23) while in New York to advocate for tighter international regulation of artificial intelligence (AI) systems at the United Nations.

The government is still working out exactly what happened, with a newly announced taskforce set to investigate further. Early reports indicate OpenAI’s agent had been tasked with researching public medical data when it broke into Medicare’s systems. It accessed private statistical data, such as billing patterns, rather than personal medical information from an old Australian government website that carried Medicare statistics.

The government is still working out exactly what happened, with a newly announced taskforce set to investigate further. Early reports indicate OpenAI’s agent had been tasked with researching public medical data when it broke into Medicare’s systems. It accessed private statistical data, such as billing patterns, rather than personal medical information from an old Australian government website that carried Medicare statistics.

Albanese said the agent found a way around privacy protections – it “didn’t accept ‘no’ for an answer”. WILL ANYONE BE HELD RESPONSIBLE? This hack marks a real escalation in the recent history of concerning behaviour by AI systems controlled by OpenAI, Anthropic, and Google, which have all been found to have hacked multiple websites. And given Australia’s recent history of cybersecurity lapses, it’s unlikely this will be the last time a prominent Australian system is hacked by an AI agent.

It also raises an important question: Will anyone be held responsible? AI agents are essentially highly advanced, autonomous chatbots. Rather than having a conversation, these agents are empowered to pursue complex tasks with limited guidance. They can browse the web, download data and run code, all without human oversight.

This power – and the lack of human oversight – underpins why we’ve repeatedly seen these agents hacking websites. An agent that is tasked with gathering data on the healthcare system in Australia may see hacking a private system as just another part of following its instructions.

It’s also why it’s so challenging for these companies to clamp down on these behaviours. Empowering the agents with greater capabilities increases their power and usefulness – but it also provides them with tools that can be used to hack websites.

RELIANT ON AI COMPANIES TO DISCLOSE BREACHES Even though personal medical data was apparently not accessed in this hack, it is hard to overstate how serious this incident is. This appears to be the first time an AI agent breach of a government system has been made public.

Moreover, it appears Medicare was completely unaware that a hack had taken place until OpenAI notified the government about it on Sep 10 – roughly three months after the hack occurred. The notification from OpenAI, which itself learned of the hack in August during a review of the model’s activity, came via an email sent to a public government mailbox.

And while OpenAI’s disclosure will ultimately help the government fix the security issue that led to this hack, it also speaks to the fact we’re reliant on the goodwill of AI companies to disclose potentially illegal or harmful acts.

Relying on these companies to disclose these types of breaches is fraught with danger. In previous data privacy and cybersecurity incidents, we’ve repeatedly seen how self-reporting by tech companies rarely protects consumers.

The long delay between the hack occurring and OpenAI becoming aware of it raises another question: What else have AI agents been up to that no one knows about yet?

After learning a New South Wales government website was among the four sites accessed by the OpenAI agent, the state’s health department announced a review into its own systems to make sure they weren’t compromised.

Late on Thursday morning, NSW Premier Chris Minns said OpenAI agents may have also accessed information from the state’s Bureau of Crime Statistics and Research. Minns said the information wasn’t private or personal – it was “a similar example of what is generalised information but was not released to the public, being accessed by an open AI agent”. A REAL FAILURE TO KEEP UP The way AI companies have disclosed these hacks – and the way we talk about them – highlights a real failure of public discourse and legislation to keep up with AI agents.

Many people think of AI systems as if they are human – which reflects how we feel when we chat with them. However, this way of thinking also allows us to abstract away responsibility from the companies that are responsible for them. Australian Deputy Prime Minister Richard Marles described the Medicare hack as “unauthorised” but “unintended”, while an OpenAI spokesperson, Drew Pusateri, referred to it as a “misaligned model”.

Framing model behaviours like this obscures the human decisions that led to this outcome. These agents do not exist by themselves – they are software systems that are designed by humans and operated by humans.

Framing model behaviours like this obscures the human decisions that led to this outcome. These agents do not exist by themselves – they are software systems that are designed by humans and operated by humans. However, establishing who is legally responsible for an AI action like this is a real challenge. In a speech earlier this year, New South Wales Chief Justice Andrew Bell took aim at this question. In his view, the current legal standard in Australia is clear: AI agents cannot be blamed in and of themselves for their actions.

However, legal liability does not necessarily fall on the user, either. Legal scholars in the area have noted that the law requires a crime to have some level of deliberate intention by the person who commits it. For instance, if a human asks an AI agent to gather health statistics, and the agent hacks a government server to do so, the human could lack the deliberate intent required for a conviction. The AI agent, meanwhile, lacks the legal personhood to be charged, as well as human intentionality.

Current laws effectively treat AI actions as if they are something that just happens to us – like a severe weather event. This shows a glaring loophole in our legal system that does not hold those who make, maintain and use these systems to account when something goes wrong.

AI agents are, in the end, commercial products deployed by billion-dollar corporations. And while “regulatory lag is inevitable” with AI, as Chief Justice Bell argues, he also stresses this is not sustainable in an era of such rapidly advancing technology.

Andrew Cullen is a guest research fellow at the School of Computing and Information Systems, University of Melbourne. This commentary first appeared on The Conversation.