Immigration Minister Brendan O'Connor has rejected expert assessment that suggests only three per cent of temporary worker visas are questionable.

But he can't nominate the actual percentage.

"I am looking at trying to precisely estimate the proportion," Mr O'Connor told Sky News on Wednesday.

The minister has announced a clampdown on the 457 temporary foreign worker visa program, saying the government will address abuses and ensure Australian workers get first preference for jobs.

But two of the government's advisers on skilled migration have criticised the move and questioned the level of rorting.

Australian National University demographer Peter McDonald, who sits on a ministerial advisory council on skilled migration, estimates about three per cent of cases are questionable.

"It is not out of control," he told The Australian Financial Review.

"On the fringes maybe the system is being rorted and some of those jobs could be filled by Australians."

Council colleague James Pearson, the West Australian Chamber of Commerce and Industry chief executive, said he understood from the immigration department that "something less than one per cent of employers have been abusing the system".

Mr O'Connor said he was seeking further information.

"But it is more widespread than has just been referred to," he said.

"We're seeing a greater level of disquiet in relation to certain businesses operating this scheme in a way that seems to exclude opportunities for local workers."

Mr O'Connor said there was evidence that wages were dropping in occupations where there was a saturation of temporary foreign workers.

"What that says to me is overseas workers are willing to come in on lesser wages which have pushed down wages at a time when wages should have been on the rise," he said.

Liberal backbencher Josh Frydenberg said the government was responding to its trade union paymasters.

"The reality is that the nurses, the engineers, the doctors and the accountants who come to Australia under 457s are doing a lot to strengthen our economy," he told Sky News.

"There has been only one successful prosecution about 457s where they were abused."

Labor MP Ed Husic said the visas were introduced following an under-investment in job and skills training by the Howard government.

"We saw all sorts of skill shortages arise," he said, adding the 457 program was a stop-gap measure in many respects.

The visas were needed for local business but the integrity of the issuing system had to be monitored.