Demonstrations at the Villawood detention centre in 2011 became so violent staff feared for their safety, a Sydney jury as heard.
Twelve men have been charged with rioting at the centre in Sydney's southwest on April 20, when three buildings burnt down.
Another man, Sayed Ibrahim Mosawi, is accused of aiding and abetting them.
On the opening day of their trial on Wednesday, Crown Prosecutor Greg Farmer SC told the NSW Supreme Court the 12 men's "common purpose was that they were protesting their detention and that they determined to do that by way of acts of violence".
He said the protest began with two men sitting on the roof of the Macquarie Building around 8am.
By 7pm, he said, a large number of detainees had broken through a cordon and made their way onto the roof and begun shouting words like "freedom".
Less than four hours later, there was "a great escalation of events", with wheelie bins and benches being destroyed and violence occurring towards people, the court heard.
Around midnight, Mr Farmer said, Serco staff at the centre decided to evacuate "for their own safety" and a burning torch was thrown through a window.
He told the jury the question was not whether a fire and riot occurred but rather whether 12 of the men before them were guilty of the charge of riot or alternatively, affray.
Mr Farmer said the case was not about whether or not they should have been detained but "who did what".
"It's a question about what happened whilst they were detained," he said.
The trial before Justice Robert Hulme continues.