Prime Minister Anthony Albanese says a decision by the Liberals to oppose a Voice to Parliament will make it difficult for a referendum to pass.