Most AI apocalypse scenarios seem predicated on the idea that the AI will be murderously insane.
But what if it's just better than you?
3
11
14
Conversation
Replying to
Imagine if you generate a sufficiently powerful AI and its reaction is not murder, but "oh my god you poor, lost things! let me help you."
1
Replying to
This of course assumes a kind of human-esque moral sentiment, but the point is that we are not so special, not so good, not so nice.
1
Replying to
And if we create a being more powerful than us, for example in an attempt to solve some major problem, and it develops its own agency...
1
Replying to
Chances are good it would try to optimize for solving our problems, possibly better than we could even imagine. Would people accept that?
2
Replying to
The possibility space of intelligence is vast, there's no reason to assume a random intelligence would care about us
1
So it would have to be explicitly designed for that. I suggest reading up on the Friendly AI problem, paperclip maximizers etc
1
"Superintelligence: Paths, Dangers, Strategies" by Nick Bostrom seems to get recommended a lot though I still haven't read it
4
Replying to
But I suspect we are more likely to see advances in AI made by accident rather than by design (as in most other hard sci fields).
Replying to
Well, why do you think chances are good it would optimize to solve our problems?
11
Replying to
To qualify that a bit, it depends on what "our problems" would be defined as in relationship with what it's been designed to do...