The former chair of the Conservatives has suggested he would find it hard to vote for his own party in this election.