The political debate over migrants hasn't turned ugly yet - but it could