Conversation

GPT-3: Language Models are Few-Shot Learners, by

et al. “We train GPT-3, an autoregressive language model with 175 billion parameters, 10x more than any previous non-sparse language model, and test its performance in the few-shot setting.” arxiv.org/abs/2005.14165

OpenAI

1:53 AM · May 29, 2020

Replying to

Fun experiment: they tested GPT-3's ability to perform simple arithmetic problems in natural language (without explicitly training it to do arithmetic)

135

hardmaru

@hardmaru

May 29, 2020

With 175B parameters, the language model can perform three digit addition and subtraction from natural language, but wasn't able to generalize beyond three digits.

124

Replying to

and

GPU go brrr

Replying to

and

Release it so we can improve the AI Meme Generator

Replying to

and

"Unfortunately, a bug in the filtering caused us to ignore some overlaps, and due to the cost of training it was not feasible to retrain the model." It is interesting the new dilemmas faced in the "train it once" regime

Replying to

and

you'd need to buy another hard drive just to store the parameters of the full model. It's +600 GB, right?

Replying to

and

I did quick back of the envelope calculation and concluded that it would cost around $2M to retrain their largest model using AWS p2.16xlarge instances. Nice.

Replying to

and

- What's up, GPT-3 ? - Bigger, Better, Not Faster, Nothing More! #musical #reference

Replying to

and

Super impressive, but I'm not very happy that the trend towards massive models continue.

Replying to

and

Hey

, I'm extremely disappointed that you guys didn't decide to include the Once and Future String pun in this paper. Would have been a double pun! (a: LMs take in once and predict future string, b: the return of GPT as King of expensive LMs)

Quote Tweet

Sam Finlayson

@IAmSamFin

May 22, 2020

Attention deep NLP people: I've got a great title for a language modeling paper -- all I'm missing is the method. It's called "The Once and Future String." Hit me up if you've got any good models and I'm happy to split the authorship 50-50 on the NeurIPS submission.