Julie Inman Grant, the eSafety commissioner, faced federal parliament's inquiry into AI on Tuesday.
The safety of all Australian internet users, particularly children, was front of mind as she urged the government to avoid playing catch-up on the technology, as it had in the past.
"We've run this experiment before," Ms Inman Grant said.
"For years, regulation could catch up with social media harms after the damage was visible in children's mental health and election integrity and trust in our information ecosystem.
"AI is moving faster and its harms could be much more catastrophic."
She noted that the regulator has already seen AI weaponised to create child sexual exploitation and abuse material, noting that there was a 75 per cent increase in reports between 2024 and 2025, particularly in schools.
In response, the commission began working with the National Home Safety Education Council, a group of 27 departments and organisations, assisting in the rollout of a school-based reporting system.
Enforcement action against several so-called "nudify" services had resulted in eight of the most widely used programs withdrawing access from Australia or introducing age restrictions and stricter standards.
The commission had a 96 per cent success rate at getting image-based sexual abuse material taken down.
But that action was only possible if there was a sexual component to the material, a gap Ms Inman Grant acknowledged.
"We're seeing image manipulation of teachers or principals doing and saying things they didn't do, which can be harmful and defamatory and hurtful to their reputations," the commissioner said.
"But without that sexualised component, we don't have powers to be able to remove that content."
Other content could only be removed if it met the high standard for adult cyber abuse, a power the commission had only been able to use successfully in six per cent of cases.
There was also a gap in regulation around the use of deepfakes in an election context, but Ms Inman Grant noted the issue fell outside the commission's remit.
Independent MP Kate Cheney raised the possibility of AI-powered chatbots being included in the social media ban for children under the age of 16.
Research found one in five young people who used AI chatbots or companions had already experienced inappropriate or harmful interactions, and one in three gave personal information away.
"The engineered sycophancy and anthropomorphism into these AI companions and chatbots make them even more insidious," Ms Inman Grant said.
"We should be as protective as we can be without undermining children's digital rights and their ability to reach out and find things."
The commission will explore how it can extend existing standards to address issues surrounding AI chatbots under proposed digital duty of care laws.
The proposals, introduced to parliament on Thursday, will put the onus on platforms - rather than users - to ensure online environments are safe/
The inquiry will also hear from the Australian Securities and Investment Commission, the national science organisation and technology experts.
A final report is due by the end of November.
