Gary Marcus

Last week: Deep learning is on verge of understanding natural language. Problem of tracking world change over time solved 3 years ago. This week: Nobody ever said statistical language models understand anything.https://twitter.com/tdietterich/status/1191756945226522624 …

Now now. I didn't say they understand nothing; you know very well I wouldn't claim that. The "extend this prompt" demos show that the new models extend the span of "local coherence" far beyond what n-gram models would capture. But they also reveal major semantic failures...

real question is whether deep learning alone can maintain representations of the world as it unfolds in a narrative / article etc. aside from highly limited worlds of the FB babi tasks, i have not seen any evidence that they can. and gave principled reasons to think otherwise.

Even on limited words babi tasks what happens when two or more tasks are clubbed together? I wud say the first sign of understanding will come when a system can pass any of the babi tasks individually or collectively with same accuracy.

That said, the babi tasks tests very limited aspect of only a few senses. Like in task 14 time manipulation, it only tests ability to understand before/after or timespans like morning or evening. Understanding time involves far more complex aspects than that.

I think babi is a very limited (i.e., nearly useless) set of tasks. But I would reword your statement to say "There are many more aspects of time to understand than just that."

Exactly my point. That the performance of state of art AI wud go nuts if one combines just two of these "nearly useless" tasks. And even when stat based AI pass this time manipulation task, they dont understand even "just that".