Kevin Simler

1/ It’s not that rationality doesn’t work; it’s that it works for reasons completely different from the ones it claims.

2/ The reason this matters is that rationality’s failure modes are not the ones rationalists expect.

3/ Rationalists’ expected failure modes: parameter uncertainty, incomplete information of known types, insufficient computation power, etc.

4/ Rationality actually works through intelligent interpretation of inherently ambiguous rules in concrete but ambiguous situations.

5/ Some typical rational failure modes: model vocabulary fails to make relevant distinctions; sensible rule misinterpreted in specific case;

6/ major aspects of circumstances were unexpectedly not accounted for by the model *at all*—it’s not even wrong, it’s entirely inapplicable;