The Prime Minister's baby brother said he had suffered "torn loyalty" in a bombshell farewell as a minister and MP