Britain cannot get access to the single market unless London also accepts the freedom of movement of workers, EU leaders said.