Gary Marcus

so what are the well-defined classes of models for which you personally publicly acknowledge limitations @roydanroy? (and which limitations).

For starters, linear regression cannot learn nonlinear relationships.

I'm a little uncomfortable saying what a Resnet-150 trained by SGD cannot do. That's a research question. Surely, it cannot do something that cannot be done with 150 steps of approximate real computation. Of course, that includes an infinite number of things.

If you add memory you get a Turing machine, and then the question is, how do you learn it? I'm a little uncomfortable saying what SGD can and cannot do. It's not a well defined model class anymore.

can SGD do the things that Davis and I say it can't?

I've no clue. But I can say something almost trivial. If you add noise to SGD and scale the learning rate, then it becomes a sampler and sampling can implement anything Bayes. So, algorithmically, we're talking about whether or not the noise in the data is sufficient.

But then you see something like this: http://www.sontaglab.org/FTPDIR/maass_sontag_analog_neural_networks_gaussian_neural_computation1999.pdf …

Yeah, so that's a great example of an actual limit. Few people add noise post activation but then again we're using floating point. When people talk informally about limits, they are not speaking about precise limits.