Trade Minister Craig Emerson says it's clear journalists are being briefed by "people who have a different view" about who should lead the Labor Party.
But the Labor cabinet member says there is no plan to topple Julia Gillard, and has reaffirmed his support for the prime minister and the "terrific" job she was doing.
Labor MPs on Monday were forced to hose down leadership speculation as details of two polls showed previously safe seats were at risk in the coming federal election.
This followed a media blitz by former prime minister Kevin Rudd last week, and a fortnight of parliamentary sittings in which Labor MPs were left gloomy about the prospects of holding government.
Dr Emerson said he was aware journalists were being approached by "people who have a different view as to who the leader should be".
"I'm not suggesting for a moment that the media makes this stuff up, it's not media speculation," he told ABC TV on Monday.
"But the fact is ... nothing is actually going on in terms of action."
He said suggestions cabinet minister Bill Shorten no longer supported the prime minister were wrong.
Mr Shorten has reaffirmed his support for Ms Gillard, amid reports he's under pressure from within the party to tell her to go.
Former Labor leader Mark Latham said if prime minister Gillard stepped aside for Mr Rudd, she would be rewarding "a saboteur".
Mr Rudd had been running a "jihad of revenge" against the Labor Party since being toppled as Labor leader in 2010, he said.
"It goes beyond the normal human reaction of revenge.
"You're getting into the realm of evil here with Rudd."
Opposition frontbencher Malcolm Turnbull said the animosity and hatred tearing the Labor Party to pieces was "frightening" and like nothing he'd seen in politics.
"Labor is a hopelessly divided, dysfunctional rabble at the moment," he said.
"It has to sort out its problems, it can only do that in opposition."