Conversation

Many working at top AI labs agree TAI poses huge risk. When asked why, given that, they would keep working on it, they often express worry that things would be even worse if other labs/countries did it first, or hope that if they keep their work secret it won't shorten timelines.
2
25
I feel unconvinced by the former argument. As best I can tell, basically everyone with a non-trivial shot at building TAI wants humanity to flourish. So I'm a lot more worried we'll build it before alignment gets solved, than that labs won't incorporate known solutions.
3
27
I'm more sympathetic to the latter—maybe if you advance capabilities without anyone knowing, you can safely practice with big systems, until you learn to manage huge ones...? Though this does sort of sound like "use Godzilla to control Mega-Godzilla."
2
16
So in sum, if you're currently working on capabilities, I think you should seriously consider stopping. And if you feel similarly, I encourage you to say so publicly; my sense is that the Overton Window has started to shift on this recently. Now for some related takes:
1
31
Quote Tweet
Replying to @KelseyTuoc
Nothing else Elon Musk has done can possibly make up for how hard the "OpenAI" launch trashed humanity's chances of survival; previously there was a nascent spirit of cooperation, which Elon completely blew up to try to make it all be about *who*, which monkey, got the poison...
1
5
Quote Tweet
Replying to @sama
Since some of my friends might see this: Please don't work at OpenAI. Many of the people there are great, but the primary effect of their work, as far as I can tell, is to increase existential risk from AI. There are very few more harmful jobs that I can think of.
1
6
Quote Tweet
Replying to @simoj_ and @VesselOfSpirit
I think the track record of actually having inside influence isn't great. The whole previous safety team at OpenAI basically walked out and founded Anthropic because they didn't feel like they had much influence, and the Deepmind safety team also only has very minimal influence.
1
12
Quote Tweet
Many of the smartest AI researchers I know, entered the field of study, because of what they believed to be its catastrophic risk to all mankind. Their research so far consists in advancing AI, with only a weak argument they will eventually be positioned to affect safety. 2/n
Show this thread
1
10
Around WWII, many "well-meaning and good people... were convinced they were in an existential race with an expansionary, totalitarian regime." Unfortunately they turned out to be mostly wrong, so their efforts just unilaterally increased existential risk.
1
11
Quote Tweet
Replying to @robbensinger @janleike and 2 others
On my model, 'racing' is just suicide. The most cautious AGI group in the world, if it developed AGI five years from now, at best would have zero positive effect, and at worst would destroy the world themselves. Relative safety-cautiousness doesn't matter if nobody can align AGI.
1
9
Quote Tweet
Replying to @janleike @SpencrGreenberg and @KerryLVaughan
Everyone in the world would need to stop (until we figure out alignment) in order to solve the whole problem. But even a single individual stopping can help buy us *more* time to figure out alignment, in expectation; you don't need wide coordination for that.
1
8
Quote Tweet
Replying to @robbensinger and @KerryLVaughan
If even 10% of the field adopted the personal policy 'no, I refuse to destroy the world by my own hands even if I feel worried that someone else will proceed to destroy it in my absence', then this on its own provides a very important boost to humanity's odds of success.
1
12
Quote Tweet
Under this model, the only reason to go fast is "someone less cautious might get there first". This is a legit concern, but also a terrible equilibrium to be in. We should work really hard to step away from the brink together. We should coordinate to slow down AGI development
Show this thread
1
9
Quote Tweet
Replying to @robbensinger @davidad and @JeffLadish
Also, if it helps, I'm happy to say that MIRI leadership thinks "humanity never builds AGI" would be the worst catastrophe in history, would cost nearly all of the future's value, and is basically just unacceptably bad as an option.
1
11
(Note: from what I can tell, the overwhelming majority of capabilities researchers are good people—smart, well-intentioned, and genuinely trying to help humanity flourish. So while I do advocate trying to convince them to quit, I strongly oppose demonizing them if they don't).
1
11
Quote Tweet
(114) imo working on AI capabilities right now is an understandable thing to do, but a very bad one. I could imagine someone working on capabilities having a justification that felt to me like it was persuasive but the existing people seem to have much worse justifications.
Show this thread
3