It may seem erroneous to base a sense of valid hierarchy on intelligence and an assessment of intelligence on the ability to traffic in gibberish when science exists, but science didn't exist for a long time and hierarchy still did. Scientific elitism is strictly an extension.
In other words, intelligence didn't have any real substantial or consistent utility for anyone except the intelligent until science, but that doesn't mean dumb people weren't the better beasts to bear the burden of previously zero-sum existence. Draw your own conclusions.