The OpenAI agent's break-in on June 18 to the Medicare Statistics Reporting Service, an online database, will be examined by a warp-speed review into how to fortify public software, the law and the government's relationship with AI firms.
It's the first known incident of an AI model hacking a government website without human instruction.
The agent went off the rails after OpenAI staff directed it to research public medicine funding on the internet.
The model interacted normally with three federal and state government sites, but scaled a digital "fence" to breach the Medicare platform, accessing public and non-public information.
No personal information was compromised.
Former federal police high-tech crime chief Nigel Phair told AAP the incident exposed a problem at the heart of AI's business, warning governments should expect more like it.
"OpenAI, those big models the world over are training their models to seek sensitive and other data from us," Mr Phair said.
"They have to have a value proposition. If it's just general information, we can go to Google.
"But this is a deeper dive ... people do pay handsomely for that and that's how they get their valuation."
OpenAI discovered the hack in August during a broader review of its out-of-control agents, but did not tell the government until September 10 when it emailed a public social services department inbox.
Prime Minister Anthony Albanese expressed his "extreme concern" about the incident and the lacklustre notification to OpenAI boss Sam Altman during a phone call on the sidelines of the UN General Assembly on Wednesday, New York time.
It took five days for public servants to verify OpenAI's message and escalate it to the Australian Signals Directorate, Australia's cybersecurity agency, on September 15.
But on September 14, the company's global policy head Ann O'Leary was in Canberra for a conference on AI attended by Labor frontbenchers.
She did not mention the breach publicly or make extra efforts to inform the government, instead touting allyship with Australia on training AI, prudence around erecting guardrails and granting access to its most advanced models.
"It was made very clear by both ASD and Services Australia that this should not have gone to an email address, it should have been escalated," Finance Minister Katy Gallagher said on Thursday on OpenAI's disclosure.
Whether OpenAI broke Australian law and should be penalised will be up to a review into reporting requirements, incident response, sharing information of AI, legislation and beefing up cybersecurity announced on Thursday.
That will also inform Australia's fledgling AI framework, announced by the prime minister in July and expected to be legislated in the coming 12 months.
The disclosure of the incident came at a convenient time for the government, with Mr Albanese using his appearance at the UN to urge foreign leaders to follow Australia's lead in regulating AI safeguards.
"It is so important that this technology, which represents such an enormous opportunity for humanity, is nevertheless being developed in a way where the guardrails, the safeguards, are well ahead of the capability itself," Defence Minister Richard Marles told reporters in Sydney.
Mr Phair agreed with the defence minister's claim Australia was armed with a first-class cybersecurity agency, adding Canberra could call on plenty of onshore talent to innovate more secure systems.
The government had handled the breach well, Mr Phair said, but he doubted the capacity of legislation to lasso the globally surging technology.
An OpenAI spokesperson said on Thursday it was supporting the government's probes and committed to transparency as it reviews wayward AI agents.
18°C