Brexit campaign is told to focus on immigration