The incident has left the church embarrassed and apologetic, with Archbishop Roger Herft saying the Reverend was now facing disciplinary action and the possibility of being removed from his position.

"I find the whole story to be quite reprehensible. There is an issue in the law, but once the owner was established the moral code must apply," Archbishop Herft said.

"There will be some explaining to be done, and I am hopeful Rev. McAuliffe regrets it deeply."

After the issue was made public in The West Australian today, the Reverend - a former lawyer - backed down from his stance and agreed to return the bracelet to the couple.

But Archbishop Herft said his actions were too little, too late and he had personally apologised to the Bevans.

"The right decision has been made but it should have been made right at the beginning," Archbishop Herft told ABC radio.