The Goulburn Valley is housing asylum seekers who have been exploited, according to Shadow Immigration and Citizenship Minister Kristina Keneally.
But Member for Nicholls Damian Drum fired back, saying Labor is talking "rubbish".
Federal Labor members, including Victorian Senator Raff Ciccone and Member for Bendigo Lisa Chesters, hosted a round-table discussion last week with members of the Victorian Farmers Federation, City for Greater Shepparton councillor Dinny Adem and trade union representatives.
Labor said more than 4000 asylum seekers had arrived in Australia since July 1. It claimed 90 per cent were not legitimate refugees.
Senator Keneally said the round table was a chance for Labor to learn more about exploitation of workers by labour hire companies in the horticulture industry.
“I'm sure it's happened, unfortunately, here in the Goulburn Valley, to people when they're isolated on farms and vulnerable to things like sexual abuse and having their passports taken off them,” the former NSW premier said.
“We heard plenty of evidence today and frustration from council, from unions, and from the growers' association of these stories being widespread and happening locally (in the Goulburn Valley).”
Although Senator Keneally said there was evidence of criminal activity, she said the solution was not just sending in police and border force officers.
“To do something that abrupt will have unforeseen consequences,” she said.
“This is more about trying to find a solution for the workforce demands of horticulture, but also to ensure that we are filling that workforce with people who are properly paid and who have access to services.
“And really, at the end of the day, people who are helping to contribute to building up the Australian economy and the community.”
Senator Keneally said this was not a new problem and in many cases it was easy for people-smugglers to bring people to Australia for up to four years.
It is the second fact-finding mission Labor has hosted, with a previous one in Canberra.
Senator Keneally said more round tables would be planned but could not confirm when or where they would happen.
Mr Drum said Labor was exaggerating the extent of exploitation in the Goulburn Valley and the politicians could not find what they were looking for.
“I think it just shows that the Labor Party is totally out of touch,” he said.
“They come over here, and actually start arguing we need to make it harder.
“It's a bit of a furphy they come into Shepparton, nobody wants to meet with them, they make a whole range of accusations, and drive out of town.
“I think these accusations have been dramatically exaggerated. The vast majority of people in Australia, particularly in the Goulburn Valley, are happy that we have our borders under control.”
Mr Drum said it was closer to the opposite of exploitation, as the region had a shortage of skilled workers.
“The biggest issue in relation to employment and migration in Goulburn Valley is that so many of our businesses are Australian-owned businesses and need a legal source of foreign labour.
“I've been advocating for a relaxing of the rule to make it easier for the Goulburn Valley to access overseas workers.
“We (the Federal Government) decide who comes into this country and who doesn't. I sit on a position slightly more aggressive when I suggest we need to make it easier for our Australian businesses to get access to overseas workers.
“The whole intent of the Labor Party is to make it more difficult, to put up a false story of worker exploitation in the Goulburn Valley; it's just not true.”
Mr Drum said if any serious exploitation was occurring, the "book must be thrown" at the perpetrators.
“There's a range of opportunities for that to happen,” he said.
“There's the ombudsman, the employment ombudsman, Fair Work Australia, unions ready to jump if anyone does the wrong thing.
“I want the book thrown at them because otherwise they'll make it more difficult for people to get access to workers into the future."