The former minister says he is "increasingly feeling politically homeless" due to his opposition to a no-deal Brexit.