An Australia Institute poll of more than 2000 people showed more than two-thirds supported giving workers on the Pacific Australia Labour Mobility (PALM) scheme the right to change jobs if they are underpaid.
About 33,000 temporary migrant workers from the Pacific Islands and East Timor currently live in Australia to fill the shortage of unskilled and low-skilled labour.
A PALM worker cannot leave their job, earns as little as $70 per week and doesn't have access to Medicare.
The scheme ensures a minimum wage but businesses working alongside the program can make deductions, Australia Institute's research manager Morgan Harrington said.
He said some of the deductions include flights to Australia, visa costs, accommodation - often provided by the employer - and transporting the workers from their housed residence to the work site.
"Once all of those deductions are made, PALM scheme workers can be left with very little in their pocket," Mr Harrington told AAP.
"There is a power imbalance."
Their payslips sometimes leave PALM workers with as little as $70 per week, Ken Dachi from Welcoming Australia said.
"I've seen horrendous stuff on payslips," Mr Dachi said.
"I have personally been to dwellings where you have three people in one room, no bathroom facilities, no kitchen, nothing.
"They're paying something in the region of $350, $360 a week... but it's such dilapidated housing."
About two-thirds of those polled also felt PALM workers should have access to Medicare and most believed workers were being housed in overcrowded or substandard accommodation.
The scheme was designed to provide work opportunities for Australia's neighbouring regions and allow for funds to be sent home.
The size of the program has drawn nine per cent of Tonga's working-age population and five per cent from Vanuatu.