EU 'drives people mad' says British PM, but Miliband wants to stay