When he called the EU referendum in 2016, then-prime minister David Cameron told voters their decision would be "irreversible".