The problem here is that you're thinking of people as atomic points and not a complex, interconnected organism. For that so called 20% to have the goodthoughts, they need a lot of people to farm, work, transport, manage institutions, etc.
Reminds me of the concept of the apex predator: for one jaguar to survive, e.g., you need many more smaller animals, and in turn a large area of jungle with thriving plants, insects, etc.
