A senior Government minister admitted it was “very difficult” for Britain to leave the European Union in five days’ time.