Members of the Love Makes a Way group were protesting about 267 asylum seekers, who may be sent back to a Nauru detention centre.
Police officers escorted the group from the office about 5pm after three hours.
Anglican Priest Father Chris Bedding was among the group that has organised similar prayer sit-ins in offices of politicians across the country.
Fr Bedding said the group urged Mr Keenan to “act justly for the lives of persecuted people seeking asylum”.
“Part of the group includes 37 babies who were born on Australian soil, so we are asking those babies be allowed to stay here and grow up in safety,” he said.
“On Nauru there is no functioning hospital and no opportunities for education.
“There’s also evidence that children have been sexually abused and there is no child protection framework on Nauru.
“We’re calling on the Government to make the call and let them stay here in Australia,” Fr Bedding said.
Mr Keenan said protesters at his office were treated courteously by staff.
“At close of business the protesters stated that they would only leave if the police were called,” he said.
“The Immigration Minister will take a compassionate approach to the 267 illegal arrivals currently in Australia receiving medical assistance and decisions will be made on a case-by-case basis,” he said.
Mr Keenan said the Government was committed to defeating people smugglers under the Operation Sovereign Borders policy.
Australian churches have expressed opposition to the Government’s asylum-seeker policies, in particular the detention of children.
More than 50 churches are calling on the Government to provide sanctuary to people seeking asylum, in fear of deportation to the South Pacific island of Nauru.