Two of the arguments I hear against building housing are “No! It will attract poor people!” And “No! It will attract rich people!” It isn’t at all clear to me which is more destructive
Probably depends entirely on the existing state of the neighborhood. Neighborhoods go through all sorts of cycles organically. What’s destructive is trying to force some top-down mandated change on a neighborhood that isn’t meant for that change.