Gary Marcus

So bottom line is - no - we are not even close to AGI right now. Even simulated conversations like GPT-2 are crappy.https://twitter.com/GaryMarcus/status/1194072212841517056 …

While the transformer is an important contribution to ML (and the implied idea of maintaining identity via attention imho relevant to AGI), GPT-2 is a text prediction tool, not an AGI architecture. The quality of a chatbot built on GPT-2 is unrelated to how close we are to AGI.

that’s retrenchment and revisionist history. few months ago people were perfectly happy to talk about gpt-2 as an understanding tool; it’s in the original paper, the title of the Bet paper, sutskekers quote to the New Yorker, etc. i have posted about this before.

I don't find this claim in the GPT-2 paper, and have not met any serious researcher who claimed that GPT-2 is able to understand text in any real sense. The NLU community may use the word "understanding" more broadly than it should, but is very aware of the limitations.

Did anyone in deep learning community call Sutskever out for equating understanding and prediction in the New Yorker? for pitching GPT-2 in orig. blog as having "rudimentary reading comprehension? Or BERT paper for having "understanding" in title? AFAIK, no.

On the one hand, “understanding” and pure NLP papers shouldn’t be in same sentence/article. On the other, lack thereof in pure NLP has ~0 to do with how close we are to AGI. In right grounded lifelong causal environment, prediction may (or may not) be key driver of understanding.