One interesting thing about the ARC competition is that it serves to highlight how people who use deep learning often have little idea of what deep learning actually does, and when they should be using it or not
Show this thread
DL is applicable when you're doing *pattern recognition*: when you have data that lies on a smooth manifold, along which samples can be interpolated. And you're going to need a dense sampling of your manifold as training data in order to fit a parametric approximation of it
-
-
Generalization in deep learning is interpolation along a latent manifold (or rather a learned approximation of it). It has little to do with your model itself and everything to do with the natural organization of your data
Show this thread -
Differentiability & minibatch SGD are the strengths of DL: besides making the learning practically tractable, the smoothness & continuity of the function & the incrementality of its fitting work great to learn to approximate latent manifold. But its strengths are also its limits
Show this thread -
The whole setup breaks down when you are no longer doing pattern recognition -- when you no longer have a latent manifold (any kind of discrete problem) or no longer have a dense sampling of it. Or when your manifold changes over time.
Show this thread -
New conversation -
-
-
Smoothness is more a requirement of gradient-based optimizers, no? Others being neuro-evolution, randomized weights, etc.
Thanks. Twitter will use this to make your timeline better. UndoUndo
-
-
-
This interpretation of DL neglects to consider applications like reinforcement learning, self play, etc. that don't rely on tons of data to learn. E.g. Finding a suction that minimizes a loss function.
-
A Solution, rather
End of conversation
New conversation -
-
Show additional replies, including those that may contain offensive content
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.