Safely aligning a powerful AGI is difficult.
Conversation
Replying to
A 'powerful' AGI is one with a sufficiently huge potential impact that we have to align it. Operationally, let's say an AGI is 'powerful' if it can invent and deploy biotechnology at least 10 years in advance of the human state of the art.
13
13
182
An 'aligned' powerful AGI is one that can be pointed in any direction at all, even what seems like a simple task that isn't morally fascinating. E.g. "Place, onto this particular plate here, two strawberries identical down to the cellular but not molecular level."
8
22
150
A 'safely' aligned powerful AI is one that doesn't kill everyone on Earth as a side effect of its operation; or as a somewhat more stringent requirement, one that has less than a 50% chance of killing more than a billion people.
13
27
204
Safely aligning a powerful AI will be said to be 'difficult' if that work takes two years longer or 50% more serial time, whichever is less, compared to the work of building a powerful AI without trying to safely align it.
14
9
168
Replying to
I don't think it's "difficult"--it's logically impossible. And undesirable. Isn't it?
In what sense would it be a "powerful AGI" if it was constrained (i.e., doomed) by the goal we programmed into it?
8
11
Show replies
Replying to
Yes if there are states of matter or being that they prefer to paperclips.
6
4
111
Show replies
Replying to
Any parallel concerns for non-general AI such as GPT-3, Midjourney, DALL-E-2, etc?
2
1
7
Replying to
Nope. If it's not smarter than you, it's not really scary.
6
3
61
Show replies



