A five-year £800m NHS out-sourcing contract ends after eight months because the company running it says it is not "financially sustainable".