But the minister in charge of the newly established body has defended its capability, saying it would have comparable features to its peers and could address risks posed by the technology.
Industry, Innovation and Science Minister Tim Ayres launched a passionate endorsement of the AI Safety Institute on Tuesday under questioning at senate estimates in Canberra.
The discussion came weeks after the federal government appointed Royal Australian Air Force AI lead Kate Conroy to serve as the institute's AI governance general manager.
But independent senator David Pocock questioned why the government did not publicly announce Ms Conroy's appointment and why the institute would receive modest funding compared to its peers in the UK and Canada.
Australia's AI Safety Institute has a budget of $29.9 million over four years, including $3.5 million in this financial year.
By comparison, the UK's AI Safety Institute has an annual budget of $120 million, and the Canadian Artificial Intelligence Safety Institute has $50 million in funding over five years.
"It seems to me that the Albanese government is really out of step with most Australians when it comes to AI and I think (their) very valid concerns about the impact on democracy, on society, on jobs," Senator Pocock said.
But Mr Ayres said the institute would fulfil one element of its three-part National AI Plan and had received "substantial investment" to monitor, test and make recommendations about AI risks and trends.
"I have no apprehensions about the capacity of this department to provide me with frank and fearless advice about these questions," he said.
"This capability will evolve over time and it will meet the challenges that Australia needs it to meet."
More staff were being recruited for the organisation, which would remain within the department of industry, science and resources, and provide advice rather than operate as a regulator, he said.
"We are carefully and deliberately making sure that capability is set up much like the UK and Canadian safety institutes, which are set up within their relevant departments," Mr Ayres said.
"They are beginning their work and they are going to get stronger and better over time."
The federal government announced plans to launch the AI Safety Institute in November, replacing an AI advisory body announced in 2024 that was expected to develop mandatory guardrails for the technology.