François CholletVerified account

The second point -- training data density -- is equally important. You will naturally only be able to train on a very space sampling *of the encoding space*, but you need to *densely cover the latent space*.

The practical implication is that the best way to improve a deep learning model is to get more data or better data (overly noisy / inaccurate data will hurt generalization). A denser coverage of the latent manifold leads a model that generalizes better.

This is why *data augmentation techniques* like exposing a model to variations in image brightness or rotation angle is an extremely effective way to improve test-time performance. Data augmentation is all about densifying your latent space coverage (by leveraging visual priors).

In conclusion: the only things you'll find in a DL model is what you put into it: the priors encoded in its architecture and the data it was trained on. DL models are not magic. They're big curves that fit their training samples, with some constraints on their structure.

I saw you were discussing this with Yoshua Bengio at the AGI conference, to which he replied that the missing piece is getting rid of the independence assumption in the latent space by assuming some additional 'modularity' prior. Do you have any comments on this?

Yoshua is right! The more priors you inject the less data you need to obtain a curve that approximates the latent manifold. Strong & accurate priors enable you to "see" further given the stepping stones (data points) you're given.

The entire subfields of DL architecture and data augmentation are about leveraging new/more priors in this way. And such priors are often about modularity! This is why we use "layers" or "convolutions" in DL instead of an amorphous soup of parameters.