Perhaps, but if we’re having this discussion, then isn’t human alignment the bigger issue?
It’s tough to expect humans to self-sacrifice for unresolved concerns of AI alignment.
Game theory suggests the right play is to accelerate, considering how humans will play it.
Conversation
It's pretty rare to find an ML researcher who thinks AGI is likely to kill everyone, but is trying to accelerate us to AGI ASAP anyway because they're worried they'll otherwise die before we reach AGI.
Far more common is one or more of:
4
3
42
- People who just avoid thinking about AGI or its implications, and tell themselves they're doing something more mundane in advancing ML SotA.
1
29
- People who tell themselves that AGI is surely far off in the distant future (because its implications sound so wild and incredible), and that there's therefore surely lots of time to think about the risks later.
Implicitly, they assume intelligence.org/2017/10/13/fir is false.
1
2
29
- People who haven't thought about AGI in any detail, let their imaginations drift toward happy thoughts regarding AGI because that's more pleasant to fantasize about, and therefore have vaguely positive model-free associations with the idea of AGI.
2
22
- People who are at least somewhat worried about AGI, but are scared to say so because they expect their peers to get mad they're lowering ML's status, or they expect their peers to think it's unscientific to talk about the future, or to dismiss AGI because it sounds like sci-fi.
2
1
21
We're killing ourselves because of mundane information asymmetries, pluralistic ignorance, and sticky Overton windows, not because of any deep conflicts of interest between anyone involved. No one stands to benefit from AGI coming sooner.
8
6
75
"Game theory suggests the right play is to accelerate, considering how humans will play it."
Why? In the situation you described, seems to me that marginal acceleration just makes things even worse. What's the supposed benefit?
3
14
If you see a row of houses on fire, and notice that your neighbors are making the fire worse, it doesn't follow that there's some amazing value to be found in your joining in to worsen the fire as well. It's not as though sooner AGI gets you much utility via *revenge*.
1
23
This Tweet was deleted by the Tweet author. Learn more
No, because rewarding humans in that situation is not the granite-sphere-maximizing action. Disassembling the humans in order to build more granite-sphere infrastructure is the granite-sphere-maximizing action in that case.
2
1
5
It's true that the AGI would probably want us to *think* that it will reward helpers. Tricking humans into helping it *is* potentially useful!
But there's absolutely no reason for it to *follow through* on such a promise.
1
5
Show replies
Why do you think that a sophisticated AI would seize on any narrow goal like that? That isn’t what real-world agents do, and it isn’t what we’ll develop our AIs to do.
2



