Presumably if you don't care for the poor/weak they will by default try to just take what they need, and violence would then be required of you want to stop them. Maybe violence
justified by some framework of principles, but still violence nonetheless.
Unless you can somehow rely on either purely passive barriers or incentives fully unconnected to physical force to sufficiently incentivize compliance. But the latter tends to be easier when people's basic needs are cared for.
Not caring for the poor / weak at societal level leads to structural violence towards them (e.g. hostile architecture, prison for those who can’t pay the bail). Physical violence is often the product of more insidious forms of violence.