Immigration minister acknowledges threat of backlash if government seen to pamper Syrian refugees