The prime minister revealed at a press conference in New York this week that an OpenAI agent scaled a digital "fence" to breach a Medicare platform and access non-public information.
Staff had only directed it to research public medicine funding on the internet.
Speaking to former prime minister Kevin Rudd, who now heads up the Asia Society, Mr Albanese said the hack was cause for concern about whether "humans remain in charge".
"This was an AI agent acting independently of its instructions, that is learning by itself," he said during an Asia Society Fireside Chat.
"So it's a bit of a wake-up call ... about the risks."
He said it showed an urgent need to regulate the technology.
"This technology is moving very very fast, and we need to make sure that we have a responsibility to keep on top of it," Mr Albanese said.
An inquiry into the hack has been established, he said.
The timing of this revelation was "pretty tactical" according to Olivia Shen, a Director at the United States Studies Centre at the University of Sydney.
"The prime minister's over there at the UN talking about the need for global good governance on AI, and we just happen to have had this incident occur that they were notified about last week," she told AAP.
The disclosure of the incident came as Mr Albanese used his appearance at the United Nations to urge foreign leaders to follow Australia's lead in regulating AI safeguards.
No matter the timing, Ms Shen said it was further proof of the need to act and enforce regulation.
"This isn't theoretical anymore ... it's urgent, and we can't keep dismissing these risks as hypothetical or the stuff of science fiction. They're literally happening now."
The breach occurred on June 18, however, OpenAI did not notify the government about the hack until September 10, after discovering it in August.
Mr Albanese said he had spoken to OpenAI boss Sam Altman to tell him he was going public about the hack as soon as possible.
Reporting and reaction times should be much quicker than the three month delay in this case Ms Shen said, backing requirements similar to cyber security incidents which impact critical infrastructure.
She said the race to develop and roll out AI to assert global dominance has also been blown out of proportion.
"There's this vision of AI as this technology that is going to be fundamentally transformative, and it probably will be. But the idea that it translates directly into geopolitical power and supremacy ... needs a lot more testing."
RMIT Innovative Interactive Technologies expert Dana McKay expressed her disappointment at OpenAI's attempt to remedy the situation by sending an email to a public Services Australia address.
"I'd be expecting them to say, 'right, this is how (the data) might have been incorporated into our models. This is what we're going to do to cleanse it from our models. This is what we're going to do to make sure it doesn't happen again,'" she told AAP.
"'Oh, and by the way, we are extremely sorry.'"
Associate Professor McKay said she was optimistic laws could be put in place to combat unsupervised agents making unsafe and illegal decisions.
"All this is happening in a climate where there is a willingness to legislate ...we might see some changes happen before it is too late."
But putting all eyes on artificial intelligence means we risk missing out on other technology breakthroughs.
"Other technologies we should potentially be looking at that are geopolitically significant. Quantum, space, synthetic biology, and biotech would be my top picks," Ms Shen said.
15°C