François CholletVerified account

In practice, we encode priors in deep learning models in two ways: 1. Architecture patterns (e.g. convolution, recurrence, attention...) 2. Data augmentation (random-yet-valid input variations) They represent assumptions about the structure of your latent manifold.

Unfortunately, the kind of prior knowledge we inject in this way is very weak compared to the complexity and unpredictability of most real-world datasets. As a result, deep learning models cannot meaningfully extrapolate except on very simple problems (special cases).

Your DL-only domestic robot won't be able to make sense of a new kitchen layout. Your DL-only self driving car won't be able to train itself in Berlin then drive in London, etc.

Your model priors can't make that jump -- not even close. They can barely move you from "here's this scene" to "here's the same scene but a bit cloudy".

(Note: you could achieve this type of local, mini-extrapolation either via a brightness normalization step or via data augmentation, both of which encode prior knowledge about the characteristics of the visual world.)

Meanwhile, a trivial example of meaningful extrapolation would be the case of a linear dataset. If your data fits on a line, *and* you make the strong (and correct) assumption that it fits on a line, then you can use your learned model to extrapolate outside of the training zone.

But in most practical settings, the answer is no, you can't extrapolate. If you want models that can extrapolate in the real world, you should move away from differentiable curves and use… discrete search over graphs of logical operators (programs). This is for another thread...