Most AI apocalypse scenarios seem predicated on the idea that the AI will be murderously insane.
But what if it's just better than you?
3
11
14
Conversation
Replying to
Imagine if you generate a sufficiently powerful AI and its reaction is not murder, but "oh my god you poor, lost things! let me help you."
1
Replying to
This of course assumes a kind of human-esque moral sentiment, but the point is that we are not so special, not so good, not so nice.
1
Replying to
And if we create a being more powerful than us, for example in an attempt to solve some major problem, and it develops its own agency...
1
Replying to
Chances are good it would try to optimize for solving our problems, possibly better than we could even imagine. Would people accept that?
2
Replying to
The possibility space of intelligence is vast, there's no reason to assume a random intelligence would care about us
1
So it would have to be explicitly designed for that. I suggest reading up on the Friendly AI problem, paperclip maximizers etc
1
"Superintelligence: Paths, Dangers, Strategies" by Nick Bostrom seems to get recommended a lot though I still haven't read it
4
Replying to
But I suspect we are more likely to see advances in AI made by accident rather than by design (as in most other hard sci fields).
1
Replying to
Well, why do you think chances are good it would optimize to solve our problems?
11
Replying to
Unless they develop totally by accident (e.g. as an emergent effect of something else), I doubt they will diverge too far too fast.