The Prime Minister says two years of 'aggressive protests' have forced the closure of his electorate office in Sydney’s inner west.