Liberal cosmopolitans, amongst whose number I suppose I should include myself, tend to warm to the idea of free movement. People should be permitted to live and work where they choose, shouldn't they? If I thought anything different I'd be open to the charge of humungous hypocrisy.
However...in the context of the EU, the theory faces some of the same headwinds in practice as the single currency. If the situation on the ground - opportunity, growth, working conditions, benefits etc. - varies between member states, and there is only a semblance of central control, then some sort of political backlash from the 'natives' (in GB's case largely former Labour voters) is almost inevitable.