Oatmeal Agitante

I have not seen one where an agent behaving catastrophically badly can't be substituted for behaving stupidly and failing

I think we don't have a good way to make agents behave intelligently in environments without fully-defined affordances

Reinforcement learning is easy in environments w fully-defined affordances, we have no idea how to do it w/o

We aren't going to put a "reward button" in one w/o, and expect an agent to "just be nice", let alone "not dirt stupid"

By "fully-defined affordances" I mean the basic building blocks of strategies for achieving the intended goal

All that said, I find that reformulating the catastrophe thought experiments to be about not being dumb is fun to think about

I get the impression the AI alignment people would basically agree; they're just trying to dereference "don't be stupid"

"stupid" from our perspective is "ruin everything for us"; "not stupid" is "make us happy"