Actually, there's a very strong argument in terms of both economics and distributive justice in favor of open borders, and I am open to it - see, e.g., @TheEconomist - https://econ.st/2AzaNNi https://twitter.com/Augustus709/status/1024737228445155330 …
I agree with you that it's used as a political cudgel, but I've always found that puzzling; an open-borders policy should not be dismissed out of hand. It's just a matter of implementation (e.g., perhaps limiting welfare benefits given to new arrivals).