Gary Marcus

3/ If you read the early papers (e.g. from @ylecun and Yoshua http://www.iro.umontreal.ca/~lisa/bib/pub_subject/language/pointeurs/bengio+lecun-chapter2007.pdf …), you see DL is a research *program* with two key elements: 1) learning is better, avoid built-in assumptions if you can, 2) use hierarchical, distributed representations trained end-to-end.

4/ Supervised training of NNs with backprop fits into that program, but does not define it. What Yoshua has been saying, as in the interview @GaryMarcus quoted, is that we want things like causal reasoning, but we want to *learn* it using distributed, hierarchical systems.

5/ My reaction to @GaryMarcus's call for hybrid systems in that essay is that he doesn't sufficiently recognize that good hybrid systems will (1) represent P and Q in a distributed fashion, (2) *learn* P and Q and the form of their relationship. That's surely how brains do it.

“Surely” seems a bit strong here — what’s your logic? I did allow in the Medium essay for embeddings (distributed codes) as a way of representing instances of variables, said that was more modern than my 2001 proposal.

I think “surely” is appropriate here actually... On (1): that the brain uses distributed codes is well established empirically and not really a Q anymore, (2) Most relational reasoning is learned. There may be some relations or concepts that are innate, but they’re limited, IMO.

2. distributed codes for motor yes; but does brain exclusively use distributed codes? why Oprah neurons

oprah neurons are not real

Quite. In fact, the data explicitly disproves any kind of grandmother cells. Consider for a moment the “Jennifer Anniston cells” that were reported in the media several years ago: