I think a measured answer has to related the degree in which an individual human's mental state could result in negative consequences. Judges (esp. SCOTUS) seem more in that category than legislators, whose activities are more collective in nature.
If presidents did not have so much singular authority over war (broad interpretations of the Commander in Chief clause and all that), I would not worry as much. But we vest a lot of potential for harm in them, as individuals. If we reduced that, then the need reduces.
