Gary Marcus

The problem with #machinelearning in a nutshell, via extensional vs intensional logic: A: {2, 4, 6, 8, 10, ... X} B: {2x: x ∈ N} There is no way for a statistical, asymbolic machine to arrive at B from A, no matter how large you choose X @GaryMarcus @filippie509 #ai

This isn't a problem with machine learning, but rather with low bias approximators. They fit the data well and can interpolate but have no mathematical reason to extrapolate. A lot of work, including in deep learning, deals with yielding better extrapolation by introducing bias.

There are many examples of this in the DL community, including work by me and colleagues on e.g. program induction https://arxiv.org/abs/1506.02516  and synthesis https://www.jair.org/index.php/jair/article/view/11172 … (aka neurosymbolic ML). Many great papers by other DL/ML peeps on these topics.

Don't get me wrong: it's great that people are pointing these things out and testing the limits of DL. But it's good to form a balanced view. Things aren't as black and white as "omg neural nets can do everything" vs "omg here's one failure mode let's ditch it all"

Agree w @egrefen here, too, but find that no matter how hard I advocate for hybrid models people always think I am arguing against all of ML... I think ML will play a huge role in AGI, but only w proper biases, including representations of operations over variables.

Properly understood, #MachineLearning is the study of the relationship of biases to what can be learned with each (and how to implement them computationally). It's bias, all the way down.

and yet innateness is a dirty word in ML. amazing.