BREXITÂ Secretary Dominic Raab has been accused of "running away" from people living in a border town.