No part of my worldview is the same after talking with Carl Shulman
Maybe 3 people in the world have thought as rigorously about so many interesting topics
Part 1 is about Carl's model of an intelligence explosion
(We ended up talking for 8 hrs, so I'm splitting this episode… Show more
MIRI
@MIRIBerkeley
MIRI exists to maximize the probability that the creation of smarter-than-human intelligence has a positive impact.
MIRI’s Tweets
Had an insightful conversation with about AI and catastrophic risks. Two thoughts we want to share:
(i) It's important that AI scientists reach consensus on risks-similar to climate scientists, who have rough consensus on climate change-to shape good policy.… Show more
216
958
3,540
RE “actual mathematical model”:
Suppose 10 million years ago, Alien A is looking at Earth, and says: “Descendants of chimps might pose an existential risk to descendants of ground sloths.”
Alien B says: “You’re crazy. They’re on different continents, adapted for different…Show more
3
2
47
So if you ask me to ignore all the particular wrong ideas that people come in with, that need refuting; and focus on your own particular interest in the particular step of the larger argument that 'alignment is superhard'; then *I* might identify these key points:
1. Alignment…Show more
12
10
64
Replying to
Thanks for replying! Your reply is lacking some detail when it comes to evaluating the proposal for predicted success or potential flaws.
I will try to fill in some system details the obvious way, and describe how the resulting system is still difficult to align. If you don't…Show more
21
22
193
AI system can generate novel proteins that meet structural design targets phys.org/news/2023-04-a via
8
20
75
To spell out the object-level argument every time, rather than just shaking my head: Most possible sufficiently intelligent minds with complicated goals, that happen not to care about humanity at all one way or another, will:
- Want to use up all of resources we use (eg, extract… Show more
51
45
397
Show this thread
Me on Bankless podcast "167 - We're Not Going to Die: Why Eliezer Yudkowsky is Wrong with Robin Hanson"
7
17
135
Bankless is an AI alignment podcast until further notice🫡
Track record so far
: 99% chance of Doom
: 50% chance of Doom
I just recorded with of He says: 95% chance of Doom
Want $ETH to $10k?
Then we gotta solve AI alignment
51
44
386
Show this thread
This is a nice demonstration of the old Yudkowsky argument of how even linear improvement can look like a dramatic transition when the human ability range is fairly narrow.
I predict it will however be more revolutionary in supplying problems than solving them.
Quote Tweet
New paper comparing GPT-3.5 & GPT-4 performance on college physics problems.
It shows that in just a few months, AI has made a leap from the 39th to the 96th percentile of human level performance.
Now imagine were it will be in 10 years.
15
30
260
Show this thread
Four mindset disagreements behind existential risk disagreements in ML: forum.effectivealtruism.org/posts/JQxvZZdP
2
3
35
Replying to
I would not have wanted to load the entire human species into one of the very first airplanes flown.
43
83
1,671
Consider that somewhere on the internet is probably a list of thruples: <product of 2 prime numbers, first prime, second prime>.
GPT obviously isn't going to predict that successfully for significantly-sized primes, but it illustrates the basic point:
There is no law saying… Show more
50
129
931
Show this thread
6
13
52
Here's my conversation with Eliezer Yudkowsky () about the danger of AI to destroy human civilization. This was a difficult, sobering & important conversation. I will have many more with a variety of voices & perspectives in the AI community. youtube.com/watch?v=AaTRHF
559
671
3,936
I think I'm moderately less excited about mech interp than I used to be, but a world that's on track to not totally drop the ball on AI has a *lot* of people who like computer-puzzle-thinking tasks throwing effort at these problems.
If you're such a person, have a look!
Quote Tweet
Thanks to @jaylbailey, 200 Concrete Open Problems in Mechanistic Interpretability now has a database! All, er, 339 problems are listed and can be sorted by difficulty. You can note which you're working on, and reach out to other people doing the same.
neelnanda.io/cop-spreadsheet
3
14
98
Nate Soares distinguishes several ways that AI systems can appear to have good goals in training, but turn out to be unfriendly in deployment:
1
4
27
Nate Soares praises for a kind of good behavior that's easy to miss: doing relatively little in recent years to worsen race conditions and drive up AI hype.
1
5
48
"Progress in AI capabilities is running vastly, vastly ahead of progress in AI alignment or even progress in understanding what the hell is going on inside those systems. If we actually do this, we are all going to die."
5
81
168
My alt/redo of the Most Important Prediction Market:
18
21
139
The average trader may not have much to contribute to it, and there needs to be a way to short-sell options, and probably the market never pays out - but this is plausibly *the* most important question on Manifold.
20
21
185
I recommend reading this post if you're thinking about how AI labs can be good and not bad for the world:
From Raymond Arnold: "In addition to technical challenges, plans to safely develop AI face lots of organizational challenges. If you're running an AI lesswrong.com/posts/thkAtqoQ… Show more
4
13
The way that the most prominent critics of AI risk totally fail to engage with even the most basic arguments made by people in the field suggests that they don't have any good counterarguments. That's very concerning!
33
58
486
Show this thread
If I wrote an "AGI ruin FAQ", what Qs would you, yourself, personally, want answers for? Not what you think "should" be in the FAQ, what you yourself genuinely want to know; or Qs that you think have no good answer, but which would genuinely change your view if answered.
309
34
467
Why doesn't everyone realize how hard it is to avoid AI doom?
It's common for scientists to race into a new field without realizing what its 50-year problems are, & become wiser with time.
The problem is that we need to get superintelligent AGI right on the first try or we die.
5
22
127
Show this thread
Eliezer was just on the Bankless podcast, emotional & honest about what's happening with AI. I recommend it.
I'm particularly touched by the segment toward the end, regarding Asilomar and the founding of OpenAI.
1
3
28
Show this thread
1
2
8
1
5
18
I'm not a big fan of the "takeoff" analogy for AGI. In real life, AGI doesn't need to "start on the ground". You can just figure out how to do AGI and find that the easy way to do AGI immediately gets you a model that's far smarter than any human. Less "takeoff", more "teleport".
13
16
173
Show this thread
Any time somebody proposes an alignment solution that says "train an LLM to output..." I suggest substituting the phrase "hire an actress to pretend..." and check if the alignment plan would still make sense.
57
40
618
Every now and then I forget that not everyone has already read this essay from 2019, which continues to be vital for thinking about LLMs.
13
41
313
Past EAs: Don't be ridiculous, Eliezer, as soon as AIs start to show signs of agency or self-awareness or that they could possibly see humans as threats, their sensible makers won't connect them to the Internet.
Reality: lol this would make a great search engine
115
402
3,864
Show this thread
2
16
Replying to
Because you need to be smarter to predict plausible plaintext from hashes, than to write down a hash followed by its plaintext. More generally, it's just not true that you only need to be as intelligent as a human to predict exactly what a particular human will say.
4
4
64
BTW, a lot of companies and projects I think are important are still pretty desperate for experienced and non-fake infosec talent, and probably will be years from now too:
3
14
52



















