Gary Marcus

That's true. But we know that it is always possible to come up with an example that can fool an ML model. To avoid this bias, we can do following experiment: test GPT for 10 random qs that are appropriate for a 5yo. We can better assess understanding capability of GPT this way?

i am building a benchmark that is very systematic, with many examples on a common theme, and dismal performance

I am confused. Is someone claiming that by ingesting lots of language, one can learn everything a typical human knows? Surely our knowledge also comes from actually being in the world and doing stuff?

if GPT-2 or BERT or T5 worked, people would be claiming exactly that.

One can reproach the transformers models of incoherence but it should be noted that the more these architectures are deep (more parameters), the more they are powerful. The latest version of GPT has an incredible performance compared to the first.

They manage to extract more and more abstraction during their training. Imagine multiplying by 1000 its scalability, what can it produce as a performance? Maybe the inconsistencies you reveal will be resolved