Gary Marcus

@tyrell_turing, I’d love to hear your opinion on this.

1/ I'll preface this small reply thread with the following: I'm responding to the linked Medium post, not @GaryMarcus's 2001 book, which I haven't had a chance to read yet (though it's on my list all to continuously expanding list).

2/ What is deep learning? Is Yoshua calling for hybrid systems that utilize non-deep, traditional symbol processing mechanisms? No, he's not. Many ppl, including @GaryMarcus, have a tendency to associate deep with its most prevalent form, but it is a far more general approach.

3/ If you read the early papers (e.g. from @ylecun and Yoshua http://www.iro.umontreal.ca/~lisa/bib/pub_subject/language/pointeurs/bengio+lecun-chapter2007.pdf …), you see DL is a research *program* with two key elements: 1) learning is better, avoid built-in assumptions if you can, 2) use hierarchical, distributed representations trained end-to-end.

4/ Supervised training of NNs with backprop fits into that program, but does not define it. What Yoshua has been saying, as in the interview @GaryMarcus quoted, is that we want things like causal reasoning, but we want to *learn* it using distributed, hierarchical systems.

5/ My reaction to @GaryMarcus's call for hybrid systems in that essay is that he doesn't sufficiently recognize that good hybrid systems will (1) represent P and Q in a distributed fashion, (2) *learn* P and Q and the form of their relationship. That's surely how brains do it.

6/ That's not to say that you don't build in some stuff - structure matters and @GaryMarcus is right that hybrid approaches are likely needed, but what form of hybrid?