Opens profile photo
Follow
Steve Omohundro
@steveom
Research Scientist for Beneficial AI.
Palo Alto, Californiasteveomohundro.comJoined June 2008

Steve Omohundro’s Tweets

5M tokens of context. Let that sink in. Yes, there's caveats. But consider what's to come: - Entire codebases in prompts - Novel-length spec docs as instructions - k-shots where k = 10K - Few-shots where each "shot" is 50K LoC → diff Those who declared the imminent death of… Show more
Quote Tweet
Meet LTM-1: LLM with *5,000,000 prompt tokens* That's ~500k lines of code or ~5k files, enough to fully cover most repositories. LTM-1 is a prototype of a neural network architecture we designed for giant context windows.
Show this thread
Embedded video
0:25
52.8K views
20
637
Show this thread
Reading OpenAI's latest paper: "Let's Verify Step by Step". The idea is so simple that it fits in one tweet: For challenging step-by-step problems, give reward at each step, instead of a single reward at the end. Basically, dense reward signal > sparse. The Process Reward Model… Show more
Image
Image
40
1,498
Show this thread
Promising. Everyone should hope that we can throw away tokenization in LLMs. Doing so naively creates (byte-level) sequences that are too long, so the devil is in the details. Tokenization means that LLMs are not actually fully end-to-end. There is a whole separate stage with… Show more
Quote Tweet
MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers abs: arxiv.org/abs/2305.07185 paper page: huggingface.co/papers/2305.07
Image
91
3,825
100k tokens. Incredible. Demos all show long input; that's just low-hanging fruit. Think novel-length instruction — entire employee manuals as prefixes. Think k-shots — embed, take 200 nearest HDBSCAN clusters, sample 3 from each. "Long prompting" changes everything.
Quote Tweet
Introducing 100K Context Windows! We’ve expanded Claude’s context window to 100,000 tokens of text, corresponding to around 75K words. Submit hundreds of pages of materials for Claude to digest and analyze. Conversations with Claude can go on for hours or days.
Show this thread
Embedded video
0:38
370.1K views
16
498
Show this thread
Our Stochastic GFlowNets paper got accepted to 😼 In this work, we propose a novel "model-based" stochastic GFlowNet method for extending GFlowNets to environments with stochastic dynamics🎲, which is essential in many control tasks.
3
87
Show this thread
Artificial Intelligence is one of the most powerful tools of our time, but to seize its opportunities, we must first mitigate its risks. Today, I dropped by a meeting with AI leaders to touch on the importance of innovating responsibly and protecting people's rights and safety.
3,902
13.4K
you people love nothing more than a "leaked internal google memo" and your breathless "no moats" retweets have compelled me to set you straight with another AI-obsessed megathread 😉🧵 tl;dr: we'll see everything, everywhere, all at once, but OpenAI (& Google) have real moats!
21
404
Show this thread
Today we’re releasing “The A.I. Dilemma” – a new talk and I gave on 3/9, a week before GPT4 launched. *Pls share it widely.* It's critical for institutions to understand how the race between AI labs is accelerating the likelihood of catastrophe:
199
1,994
Show this thread
Guys I would like to emphasize that A Game of Thrones is ~894,000 tokens The entire Harry Potter series is ~2.5M Forget asking an LLM to write a blog post or an essay Have it write you an entire high-fantasy novel series We are on the cusp of infinite, personalized media
Quote Tweet
Scaling Transformer to 1M tokens and beyond with RMT Recurrent Memory Transformer retains information across up to 2 million tokens. During inference, the model effectively utilized memory for up to 4,096 segments with a total length of 2,048,000 tokens—significantly exceeding… Show more
Image
6
104
Show this thread
## Can open-source LLMs detect bugs in C++ code? No: ``` LLaMa 65B (4-bit GPTQ) model: 1 false alarms in 15 good examples.  Detects 0 of 13 bugs. Baize 30B (8-bit) model: 0 false alarms in 15 good examples.  Detects 1 of 13 bugs. Galpaca 30B (8-bit) model: 0 false alarms in 15… ~~~~~~~~~~ hf3f8e3a 992bba08-8399-4bde-ab97-c1305e64876 SSR-I18N f2c6ac64-eb07-4bf8-bb18-52a36cf153b7 hf3f8e3a ~~~~~~~~~~
42
1,065
Show this thread