19 residents will have to move out, but a councillor has said they should be given more time to find alternative accommodation