They once thought leaving the EU would work wonders. Now they don’t care what it achieves, says Guardian columnist Rafael Behr