I think it's generally bad to work on AI capabilities. Not always (since alignment tech is often dual-use), and not forever (since never building TAI seems awful). But broadly, I think we should not build extremely powerful AI systems before we figure out how to make them safe.
Conversation
Many working at top AI labs agree TAI poses huge risk. When asked why, given that, they would keep working on it, they often express worry that things would be even worse if other labs/countries did it first, or hope that if they keep their work secret it won't shorten timelines.
2
3
25
I feel unconvinced by the former argument. As best I can tell, basically everyone with a non-trivial shot at building TAI wants humanity to flourish. So I'm a lot more worried we'll build it before alignment gets solved, than that labs won't incorporate known solutions.
3
1
27
I'm more sympathetic to the latter—maybe if you advance capabilities without anyone knowing, you can safely practice with big systems, until you learn to manage huge ones...? Though this does sort of sound like "use Godzilla to control Mega-Godzilla."
2
16
So in sum, if you're currently working on capabilities, I think you should seriously consider stopping. And if you feel similarly, I encourage you to say so publicly; my sense is that the Overton Window has started to shift on this recently.
Now for some related takes:
1
2
31
"Imagine if oil companies and environmental activists were both considered part of the broader 'fossil fuel community'. Exxon and Shell would be 'fossil fuel capabilities'; Greenpeace and the Sierra Club would be 'fossil fuel safety'..."
1
23
Quote Tweet
Replying to @KelseyTuoc
Nothing else Elon Musk has done can possibly make up for how hard the "OpenAI" launch trashed humanity's chances of survival; previously there was a nascent spirit of cooperation, which Elon completely blew up to try to make it all be about *who*, which monkey, got the poison...
1
5
Quote Tweet
Every day an AGI engineer at OpenAI or DeepMind shows up to work and tries to solve the current bottlenecks in creating AGI, we lose just a little bit of time.
Every day they show up to work, the odds of victory get a little bit lower.
My very bold take is that THIS IS BAD
Show this thread
1
6
Quote Tweet
Replying to @sama
Since some of my friends might see this: Please don't work at OpenAI. Many of the people there are great, but the primary effect of their work, as far as I can tell, is to increase existential risk from AI. There are very few more harmful jobs that I can think of.
1
6
Quote Tweet
Replying to @simoj_ and @VesselOfSpirit
I think the track record of actually having inside influence isn't great. The whole previous safety team at OpenAI basically walked out and founded Anthropic because they didn't feel like they had much influence, and the Deepmind safety team also only has very minimal influence.
1
12
Quote Tweet
Many of the smartest AI researchers I know, entered the field of study, because of what they believed to be its catastrophic risk to all mankind.
Their research so far consists in advancing AI, with only a weak argument they will eventually be positioned to affect safety.
2/n
Show this thread
1
10
Around WWII, many "well-meaning and good people... were convinced they were in an existential race with an expansionary, totalitarian regime." Unfortunately they turned out to be mostly wrong, so their efforts just unilaterally increased existential risk.
1
1
11
Quote Tweet
Replying to @robbensinger @janleike and 2 others
On my model, 'racing' is just suicide. The most cautious AGI group in the world, if it developed AGI five years from now, at best would have zero positive effect, and at worst would destroy the world themselves. Relative safety-cautiousness doesn't matter if nobody can align AGI.
1
1
9
Quote Tweet
Replying to @janleike @SpencrGreenberg and @KerryLVaughan
Everyone in the world would need to stop (until we figure out alignment) in order to solve the whole problem. But even a single individual stopping can help buy us *more* time to figure out alignment, in expectation; you don't need wide coordination for that.
1
8
Quote Tweet
Replying to @robbensinger and @KerryLVaughan
If even 10% of the field adopted the personal policy 'no, I refuse to destroy the world by my own hands even if I feel worried that someone else will proceed to destroy it in my absence', then this on its own provides a very important boost to humanity's odds of success.
1
12
1
13
Quote Tweet
I think building AGI, or making progress towards AGI, is dangerous, and people shouldn't do it.
It burns our collective commons of buffer-time between today and possible enormous destruction that we are unprepared to deal with.
Show this thread
1
5
Quote Tweet
I currently think most of the work OpenAi does has astronomically bad negative externalities and comparatively tiny positive internalities.
Show this thread
1
6
Quote Tweet
Sci-Fi Author: In my book I invented the Torment Nexus as a cautionary tale
Tech Company: At long last, we have created the Torment Nexus from classic sci-fi novel Don't Create The Torment Nexus
1
6
"The researchers at DeepMind ought to be dissuaded or discouraged from continuing to kill everybody, in addition to and in conjunction with efforts to align AI... [if pessimistic] you should be spending your time figuring out ways to slow it down."
1
6
Quote Tweet
Under this model, the only reason to go fast is "someone less cautious might get there first". This is a legit concern, but also a terrible equilibrium to be in. We should work really hard to step away from the brink together. We should coordinate to slow down AGI development
Show this thread
1
9
Quote Tweet
Replying to @thegaiaparty and @sama
There's precedent for scientists coordinating to create a moratorium. Example: en.wikipedia.org/wiki/Asilomar_
Also, you don't need to stop everyone indefinitely. You need to give researchers more time to ensure that the technology can be developed safely.
2
12
Quote Tweet
Replying to @robbensinger @davidad and @JeffLadish
Also, if it helps, I'm happy to say that MIRI leadership thinks "humanity never builds AGI" would be the worst catastrophe in history, would cost nearly all of the future's value, and is basically just unacceptably bad as an option.
1
11
(Note: from what I can tell, the overwhelming majority of capabilities researchers are good people—smart, well-intentioned, and genuinely trying to help humanity flourish. So while I do advocate trying to convince them to quit, I strongly oppose demonizing them if they don't).
1
11
"history offers a surprising array of cases where strategically promising technologies were delayed, abandoned, or left unbuilt, even though many at the time perceived their development as inevitable."
1
1
10
Quote Tweet
Do not publish capabilities research. There is sometimes a case for doing it, there is only far more rarely a valid justification for publishing it. twitter.com/scholl_adam/st…
1
5
Quote Tweet
(114) imo working on AI capabilities right now is an understandable thing to do, but a very bad one. I could imagine someone working on capabilities having a justification that felt to me like it was persuasive but the existing people seem to have much worse justifications.
Show this thread
3











