The cancellation of contracts for extra Brexit ferry services could cost the taxpayer more than £50m.