Peaceful protests calling for a change to the January 26 Australia Day holiday turned ugly today as police clashed with protesters with a number arrested.

The number of arrests could have been far higher but for a last minute pact struck between NSW Police and the organisers.

Thousands gathered for Invasion Day protests in Australia's capital cities and in regional centres. In total, upwards of 50,000 people may have marched according to estimates.

Crowds were told that when icons like Cricket Australia refuse to mark Australia Day "the walls are going to come crumbling down".

In Sydney, one speaker at an Invasion Day rally demanded $1 million for each and every indigenous Australian. Others said changing the date of Australia Day was not enough and the whole celebration should be axed.

The date of Australia Day, January 26, remains controversial as it marks the day the First Fleet sailed into Sydney Cove in 1788 formally declaring the land as British despite indigenous peoples having lived on the land for centuries.

In Canberra, a man wearing a Make America Great Again cap and waving an Australian flag was forcibly removed from an Invasion Day rally by three men in bikie colours. As he drove away, the assembled crowd cheered.

In Melbourne, thousands marched from Parliament House down Bourke Street after a peaceful protest in which police refused to remove their hats, a stance in line with police procedure. An estimated crowd of some 20,000 filled the streets of Brisbane's' CBD.

The main Sydney Invasion Day event passed off without incident with a socially distanced crowd of at least 3000 baking on a 38C day. Later, a breakaway group left the Domain as the speeches wrapped up and walked towards nearby Hyde Park.

Four people were arrested following scuffles with police in Hyde Park; two for breaching COVID restrictions; one man for assaulting police and one woman for hindering police in the execution of duty. Two other men were each fined $1000 and released.

Just yesterday, NSW Police Minister David Elliot has said the gathering was likely to breach public health orders.

"Unfortunately, anyone who attends will be exposed to fines and imprisonment," he told radio station 2GB.

Police had been able to issue on-the-spot fines upwards of $1000 but the penalty for breaching public health orders comes with a fine up to $11,000 and a six-month jail term.

With at least 3000 people in Sydney, at least 2500 could have been arrested if they didn't disperse.

But that didn't happen largely due to an agreement with police struck as the rally was being held, organiser Ian Brown, a Gomeroi man from Moree, told news.com.au.

"They allowed us to occupy the Domain and for the event to go ahead so long as there was a no marching, so that was the compromise.

"The police, given the numbers, were very cooperative so we thank them for that,"

Assistant Commissioner Michael Willing later confirmed the deal.

"We were able to move a large number of people in and out of that area, let them talk about the issues at their heart, and then dissipate in a way which was as safe as possible," he said.

WILD DEMAND OF $1M FOR EACH INDIGENOUS AUSTRALIAN

Among the protesters in Sydney was Paul Silva, the nephew of David Dungay Jr, a Dunghutti man from Kempsey, who died in prison custody in 2015.

"I'm here to demand the abolishment of Australia Day," he told news.com.au.

"Over 200 years ago the First Fleet come in and murdered, raped and stole children of our ancestors."

Mr Silva said "changing the date to is not going to make a difference".

"That we allow Australia to celebrate a day when murders and criminal activity took place is just appalling," he said.

One of the more eye opening demands from the many speakers at the event was that reparations should be paid to all indigenous Australians.

Gwenda Stanley, a Gomeroi woman from Moree suggested a figure: "A million dollars for each black person".

"Don't be fooled by the Uluru statement from the arse. Let's do reparations before treaty. A million dollars for each black person and than we can talk treaty."

Another speaker said it was clear the future of Australia Day was in doubt.

"Your citizens are waking up and our black voices are stating to resonate," she said.

"If your icons like Cricket Australia can refuse to celebrate Australia Da, y then these walls are starting to crumble down.

"The time is up, and that's why you continue to harass us."

Earlier, in the Sydney suburb of Newtown, a fresh mural painted by acclaimed street artist Scott Marsh emerged at first light.

It showed Scott Morrison dressed as Captain James Cook next to two words, "Captain Cooked", and the hashtag #ChangeTheDate.

The coronavirus pandemic this year saw Victorians unable to gather for an Australia Day rally because it was deemed a public health risk by the state government. But Melbourne City Council did approve an Invasion Day Dawn Service.

Melbourne Lord Mayor Sally Capp said the seated, 250-capacity service at Kings Domain was "a way of supporting an event that reflects that ancient Australian history".

In Sydney, organiser Ian Brown said the Uluru Statement from the Heart which proposed a voice to parliament, was not the answer.

"The statement doesn't do enough. They have this idea the statement is a grassroots movement. There was no consultation done on my homelands.

The Invasion Day rallies called for, among other things, a changing of the date to reflect the fact that for some it represents more than the beginning of British colonialism.

But there are wider aims for some. The axing of the entire celebration and a fundamental reshaping of Australia.

"We're talking about proper clan treaties; self determination and recognising everyone's sovereignty, not just people who had a continuous connection to their language and culture," Mr Brown told news.com.au.

Lidia Thorpe, the first Indigenous woman in Victorian parliament used her platform to call for change.

On Twitter, she wrote: "Too many Australians still think January 26 is a day of celebration, but for Aboriginal people across this country, it's a Day of Mourning.

"That's why I'm inviting communities, councils and organisations to fly the Aboriginal flag at half-mast on #InvasionDay."

Invasion Day protests have been planned for Sydney, Melbourne, Brisbane, Canberra, Perth, Adelaide, Darwin, Hobart, Newcastle, Rockhampton, Lismore, Albury and Lithgow.