Kevin Simler

to me this is one of your clearest expressions of how rationalism falls short

I was about to say it’s the clearest example of how he doesn’t seem to get it.

“Why do you spend so much time fixing bugs instead of concentrating on all the times the software works normally?”

Perhaps a better analogy: why are you worried about how the boiler behaves if it gets overpressurized? Why not just focus on how well it runs the rest of the time?

I don't really get this analogy. As I understand David, he's saying that rationalism focuses on how to maximize EV at explicit decision points, but ignores how to deal with facets of the system that don't show up as explicit choices

It's not [rational = bugs] vs. [metarational = working smoothly], but rather [rational = choice points] vs. [metarational = total behavior of the system]

FWIW, the main criticism of metarationalism that resonates with me is: anything that meta-R points out can then (by virtue of being pointed out) get folded neatly into R. Rationalists absolutely care about all the things metarationalists care about! So where's the meta??

I’m using “rational” to mean roughly “technical” or “formal” rationality specifically, and not jus “good thinking.” (This choice of usage may cause confusion although it’s reasonably standard and I stress the specificity frequently.)