Conversation

I don’t find this argument in itself compelling at all, since many other huge problems were solved in the past without an early clear sense of direction. I am more compelled by the arguments about why this time is different (speed, low barrier to entry, etc)
2
9
I found this to be a convincing argument that the probability is not 50/50 but not more. The last paragraph seems like an entirely different claim that isn't really argued here (though I've seen it (partially) addressed in many other places).
1
My intuition is that MIRI's argument is almost more about sociology than computer science/security (though there *is* a relationship). People won't react until it is too late, they won't give up positive rewards to mitigate risk, they won't coordinate, the govt is feckless, etc.
And that's a big part of why it seems overconfident to people, bc sociology is not predictable, or at least isn't believed to be.
1
7
Yeah I agree. The argument seems twofold of either we won't get lots of smaller failures or that society won't properly react. IMO we will get smaller failures and there will be reactions eventually. (Lawsuit is an example reaction and those are already starting)
1
Show replies
In the original text it’s clear that Scott is *caricaturing* fast takeoff and himself disagrees. I think that kind of confusion in the debate happens a lot! Eg people taking paperclip maximizes as a literal scenario instead of a random thought experiment EY made up one day.
Image
1
8
Show replies
I think that's a part of it (especially due to their belief that few people understand the difficulty) but it's also the scale of the challenge & having to get it right the first time. But I have another 40+ very long blogposts to read & digest until I can confidently say that
1
1