2/ more concretely, we find, on downstream word analogy task, it beats 1) the NTK limit and 2) all finite-width models, whose perf approaches the infinite-width perf from below as width increases.pic.twitter.com/FJ8QVsGKOO
Pokaż ten wątek
1/ Existing theories of neural networks (NN) like NTK don't learn features so can't explain success of pretraining (e.g. BERT, GPT3). We derive the *feature learning* ∞-width limit of NNs & pretrained such an ∞-width word2vec model: it learned semantics! https://arxiv.org/abs/2011.14522 pic.twitter.com/1VJVFjY7JZ
-
-
-
3/ Same thing for few-shot learning on Omniglot via MAML. IMO this is the right kind of infinite-width neural network to study!pic.twitter.com/Zbc0npnHiq
Pokaż ten wątek -
4/ This paper is the 4th in the Tensor Programs series (following https://arxiv.org/abs/1910.12478 https://arxiv.org/abs/2006.14548 https://arxiv.org/abs/2009.10685 ). In fact, I started this series 2 years ago precisely so I could eventually write this paper! I'm so happy and relieved this is now out :)
Pokaż ten wątek -
5/ Thanks to
@ZeyuanAllenZhu@prfsanjeevarora@BachFrancis@yasamanbb@LenaicChizat@deepcohen@yaringal@QuanquanGu Bobby He, Jiaoyang Huang, Arthur Jacot,@hoonkp@jasondeanlee Zhiyuan Li Etai Littwin@2prime_PKU Song Mei@ARomanNovak@vinaysrao Michael Santacroce@sschoenholzPokaż ten wątek -
6/ Lisa Schut
@jaschasd@murefil Denny Wu, Huishuai Zhang, Pengchuan Zhang for discussions and feedback! Also@roydanroy this is what I promised for "beyond NTK" :)Pokaż ten wątek -
7/ Shoutout to my co-author, former Microsoft AI Resident
@edwardjhu (who graduated into a full time position at Microsoft). Fast learner, hard worker, curiosity driven, great communicator, trustworthy - what a monster of a researcher. Follow him now! (and more to come from us :)Pokaż ten wątek
Koniec rozmowy
Nowa rozmowa -
-
-
Can we not "explain" something like BERT/GPT3* by just computing the posterior of the NNGP conditioned on all the data that GPT3 got, and then use any new data to further update the posterior? I would guess most of GPT3's performance can be explained that way
-
*GPT3 isn't even "pre-training" in the usual sense of "for later fine-tuning". It's just evaluated and it does "metalearning" on the function from inputs to outputs it implements
- Pokaż odpowiedzi
Nowa rozmowa -
-
Pokaż pozostałe odpowiedzi, również te, które mogą zawierać nieodpowiednie treści
Wydaje się, że ładowanie zajmuje dużo czasu.
Twitter jest przeciążony lub wystąpił chwilowy problem. Spróbuj ponownie lub sprawdź status Twittera, aby uzyskać więcej informacji.