New open source Flan-UL2 20B checkpoints :)
- Truly open source 😎 No forms! 🤭 Apache license 🔥
- Best OS model on MMLU/Big-Bench hard 🤩
- Better than Flan-T5 XXL & competitive to Flan-PaLM 62B.
- Size ceiling of Flan family just got higher!
Blog:
Follow
Click to Follow santiontanon
Santiago Ontañón
@santiontanon
Santiago Ontañón’s Tweets
Data on the intellectual contribution to AI from various research organizations.
Some of organizations publish knowledge and open-source code for the entire world to use.
Others just consume it.
210
430
2,116
RIP Roger Schank, who (for better or worse) shaped early perspectives of AI researcher
facebook.com/chris.riesbeck
1
3
15
Tired of tokenizers/subwords? Check out PIXEL, a new language model that processes written text as images📸
“Language Modelling with Pixels”
📄 arxiv.org/abs/2207.06991
🧑💻github.com/xplip/pixel
🤖huggingface.co/Team-PIXEL/pix
by me
30
263
1,059
Show this thread
Today, 4pm Eastern. Join us!
Quote Tweet
"Can a large language model be conscious?
Can a #LLM think?"
Join us
Fri Jan 20 4 PM ET
at the #LearningSalon
to listen to & discuss with prominent philosopher of mind @davidchalmers42
@blamlab @MelMitchell1 @csuncodes & I are excited to discuss w David!
crowdcast.io/e/learningsalo
1
4
30
#nlphighlights 138: Compositional generalization in NN models for language with Najoung Kim! We covered datasets, recent results, and how pretraining makes things complicated. Thanks , for the great discussion!
7
28
New year, new release 🔥 Introducing the first beta of openrlbenchmark, a tool to grab tracked metrics from popular RL libraries, such as SB3, CleanRL, baselines, Tianshou, etc.
💾Colab: colab.research.google.com/github/openrlb
📜Release note: github.com/openrlbenchmar
Thread🧵👇
3
29
119
Show this thread
🦔 New preprint 🦔
Lots of work has been done in the compositional generalization space recently, using tests such as SCAN and COGS. Many models actually do achieve impressive performance, some of them almost perfect lexical generalization on COGS. (1/n)
3
18
92
Show this thread
We have released "Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints"!
Our method converts a pretrained dense model into a MoE by copying the MLP layers and keeps training it, which outperforms continued dense training.
arxiv.org/abs/2212.05055 (1/N)
11
94
388
Show this thread
Given furor over ChatGPT, time to remind people about the #longbet between and Kurzweil: “By 2029 no computer or "machine intelligence"will have passed Turing Test.”
Before voting in poll (next tweet), see strict terms for the debate: longbets.org/1/#adjudicatio
#chatgpt
9
17
50
Show this thread
Applications for the first-ever Google PhD Fellowships for students in Latin America open today, along with applications to support early-career professors through Research Scholar. Read more about our investments in the Latin American research ecosystem ↓
6
78
153
Sad to see the demo of Meta AI GALACTICA taken down. The few things I tried it with got amazing outputs. It was actually useful!
1
8
🔥 CleanRL's paper has been accepted to ! Introducing at v1.0.0! We have added reworked documentation, JAX support, hyperparameter tuning, and more.
📜 Paper: jmlr.org/papers/v23/21-
💾Release:
11
38
188
Show this thread
I have a funded Ph.D. position to fill this upcoming year in the areas of:
- Interactive Narrative
- Gameplay Algorithms
- Machine Learning
- Intelligent Tutoring Systems
through the Computing and Information Sciences program at RIT. The priority application deadline is Dec 31.
3
16
17
Show this thread
10 years of FTL: The making of an enduring spaceship simulator
3
19
42
Introducing UL2, a novel language pre-training paradigm that improves performance of language models across datasets and setups by using a mixture of training objectives, each with different configurations. Read more and grab model checkpoints at goo.gle/3euHrEo
GIF
read image description
ALT
19
211
739
🤯 85% median human normalized score, 57 Atari games, 3 seeds, finished in 6 GPU days! now has one of the fastest PPO implementations in ALE w/ EnvPool and JAX. It could even rival SEED RL's R2D2 in the first 45 mins (*)
📜 docs: docs.cleanrl.dev/rl-algorithms/
A thread 🧵
5
22
150
Show this thread
Name a more iconic duo than paper deadlines and \vspace{-1em}
6
5
117
Return to Monkey Island is finally out!
Quote Tweet
Review: Return to Monkey Island is must-play point-and-click brilliance arstechnica.com/gaming/2022/09 by @samred
1
Hi everyone - I am planning to graduate by May 2023 & now looking for a full-time position in ML research/engineering!
I love reinforcement learning and open source ❤️. Please reach out and share!
CV: files.costa.sh/Costa-Huang-Re
6
22
131
Check out our new Google Research #YouTube channel for exciting new interviews, explainer series, and spotlights about how we advance the state of the art across multiple disciplines ↓
7
64
209
"Scaling laws vs Model Architectures" from .
Lessons:
- Not all arch scale the same way.
- Vanilla Transformer does pretty well 😀
- Touching the attention too much is "dangerous". 😔
- Perf at base may not translate to large+ scale.
pdf: arxiv.org/abs/2207.10551
18
239
1,001
Show this thread
Ha! Very nice! We were just talking about trying precisely this a few weeks ago!
Quote Tweet
Tired of tokenizers/subwords? Check out PIXEL, a new language model that processes written text as images
“Language Modelling with Pixels”
arxiv.org/abs/2207.06991
github.com/xplip/pixel
huggingface.co/Team-PIXEL/pix
by @rust_phillip @jonasflotz me @esalesk @mdlhx @delliott
“Language Modelling with Pixels”
arxiv.org/abs/2207.06991
github.com/xplip/pixel
huggingface.co/Team-PIXEL/pix
by @rust_phillip @jonasflotz me @esalesk @mdlhx @delliottShow this thread
1
6
I am happy to share that now has a benchmarked DDPG + JAX implementation that is roughly 2.5-4x faster than DDPG + .
📜 docs: docs.cleanrl.dev/rl-algorithms/
4
24
108
Show this thread
New software release: mctslib, a library for Monte Carlo Tree Search (MCTS) and variants. Implemented in C++ but primarily intended to be used from the Python bindings. Environments can be both defined and searched over from Python.
1
6
42
Show this thread
ICYMI: a discussion between Nobel laureate Daniel Kahneman and me from last December about human cognition and machine intelligence.
Discussing many topics like the necessity of learning world models.
Moderated by Alex Kantrowitz.
2
55
277
Language models have shown strong performance on a variety of #NLU tasks but are weaker at solving tasks that involve quantitative reasoning. Learn how #Minerva uses step-by-step reasoning to achieve a new state of the art on quantitative reasoning tasks→ goo.gle/3yGpTN7
GIF
read image description
ALT
8
121
442
Up to $100k prize for identifying an important task where large language models show inverse scaling (getting worse as they grow in size). Quite interested to find out which tasks show this effect!
6
Show this thread
, , and I are running an AIIDE workshop on AI for strategy games! You can find the cfp and more info here: skatgame.net/mburo/aiide22w The deadline is July 29 and we are accepting papers (at most 7 pages). Please RT and consider submitting!
#aiide22
7
8
Sad day today...
Quote Tweet
Vangelis, composer of Chariots of Fire and Blade Runner soundtracks, dies aged 79 theguardian.com/music/2022/may
3
Art by Renato Casaro for The NeverEnding Story (1984)
25
312
2,018
cleanrl: High-quality single file implementation of Deep Reinforcement Learning algorithms with research-friendly features (PPO, DQN, C51, DDPG, TD3, SAC)
Lang: Python
⭐️ 786
Author:
#MachineLearning
14
27
In rare interview, Monkey Island designers tell Ars about long-awaited Return arstechnica.com/gaming/2022/04 by
1
5
23
I recently moved to the Google NYC office, so, if any of you New York friends wants to grab dinner or a drink one day, let me know!
1
12
We are currently working on a new game called Dracula for ColecoVision. There's about 20 rooms already done and we're planning to have around 50 different ones in total.
Dracula is going to be the most ambitious ColecoVision game ever made, without using ANY special hardware.
146
1,085
5,049
With LongT5 we were able to scale both model size thanks to the T5 architecture, as well as input length, to get benefits from both and achieve SOTA results in several summarization tasks.
1
5
Show this thread