Ok, so, consider MNIST digits, 28x28 black & white images. You could say the "encoding space" of MNIST has 28 * 28 = 784 dimensions. But does that mean MNIST digits represent "high-dimensional" data?
Show this thread
-
The manifold hypothesis posits that for many problems, your data samples lie on a low-dimensional manifold embedded in the original encoding space.
Show this thread
A "manifold" is simply a lower-dimensional subspace of some parent space that is locally similar to a linear (Euclidian) space. (i.e. it is continuous and smooth). Like a curved line within a 3D space.
-
-
When you're dealing with data that lies on a manifold, you can use *interpolation* to generalize to samples you've never seen before. You do this by using a *small subset* of the latent space to fit a *curve* that approximately matches the latent space.pic.twitter.com/qkcmPHOaWX
Show this thread -
Once you have such a curve, you can walk on the curve to make sense of *samples you've never seen before* (that are interpolated from samples you have seen). This is how a GAN can generate faces that weren't in the training data, or how a MNIST classifier can recognize new digits
Show this thread -
If you're in a high-dimensional encoding space, this curve is, of course, a high-dimensional curve. But that's because it needs to deal with the encoding space, not because the problem is intrinsically high-dimensional (as mentioned earlier).
Show this thread -
Now, how do you learn such a curve? That's where deep learning comes in.
Show this thread -
But by this point this thread is LONG and the Keras team sync starts in 30s, so I refer you to DLwP, chapter 5 for how DL models and gradient descent are an awesome way to achieve generalization via interpolation on the latent manifold.https://www.manning.com/books/deep-learning-with-python-second-edition?a_aid=keras&a_bid=76564dff …
Show this thread -
I'm back, just wanted to add one important note to conclude the thread: deep learning models are basically big curves fitted via gradient, that approximate the latent manifold of a dataset. The *quality of this approximation* determines how well the model will generalize.
Show this thread -
The ideal model literally just encodes the latent space -- it would be able to perfectly generalize to *any* new sample. An imperfect model will partially deviate from the latent space, leading to possible errors.
Show this thread -
Being able to fit a curve that approximates the latent space relies critically on two factors: 1. The structure of the latent space itself! (a property of the data, not of your model) 2. The availability of a "sufficiently dense" sampling of the latent manifold, i.e. enough data
Show this thread -
You *cannot* generalize in this way to a problem where the manifold hypothesis does not apply (i.e. a true discrete problem, like finding prime numbers).
Show this thread -
In this case, there is no latent manifold to fit to, which means that your curve (i.e. deep learning model) will simply memorize the data -- interpolated points on the curve will be meaningless. Your model will be a very inefficient hashtable that embeds your discrete space.
Show this thread -
The second point -- training data density -- is equally important. You will naturally only be able to train on a very space sampling *of the encoding space*, but you need to *densely cover the latent space*.
Show this thread -
It's only with a sufficiently dense sampling of the latent manifold that it becomes possible to make sense of new inputs by interpolating between past training inputs without having to leverage additional priors.pic.twitter.com/SmRvEN2NXS
Show this thread -
The practical implication is that the best way to improve a deep learning model is to get more data or better data (overly noisy / inaccurate data will hurt generalization). A denser coverage of the latent manifold leads a model that generalizes better.
Show this thread -
This is why *data augmentation techniques* like exposing a model to variations in image brightness or rotation angle is an extremely effective way to improve test-time performance. Data augmentation is all about densifying your latent space coverage (by leveraging visual priors).
Show this thread -
In conclusion: the only things you'll find in a DL model is what you put into it: the priors encoded in its architecture and the data it was trained on. DL models are not magic. They're big curves that fit their training samples, with some constraints on their structure.
Show this thread
End of conversation
New conversation -
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.