Top-level takeaway of tonight's catch-up reading on ML...
I still find it hard to take AI Risk seriously as a special problem here. There are definite risks but they don't seem qualitatively different from other kinds of engineering risk.
Conversation
Replying to
Most safety engineering seems to focus on one (or a small enumerable set) of standard functioning modes, and verifies all parts are rated for that amount of eg torque. AI safety looks more like ensuring humanity is rated for a function space over the reals
1
Replying to
Climate risk, nuclear arms control, are just 2 examples that also look like that. More so in fact.
1
1
Replying to
That’s why I’m also fairly concerned about those—very complex systems with many cases to check. But 1) AI is this but more so, and 2) we spend billions every year on those two but can list on two hands all AI safety researchers
1
Replying to
That's my point. We actually spend a lot more if you classify right. All infosec research is also "AI risk" research if you conceptualize correctly without anthropomorphizing for example. If you buy an arbitrary religious boundary, there's few people inside it.
Personally I look at AI risk research as consisting of robustness and/or alignment work (infosec is robustness) that can scale to ML-powered systems that are stronger, faster, and harder to understand than today’s (a lot of infosec work won’t).
2
there's a bunch of work that would be useful to mitigate the risk of AI related accidents (e.g. formal verification, infosec), but much less happening that's directly aimed at the big, less understood problems (e.g. value alignment)



