François CholletVerified account

Great question! Yes, theoretically you can use a learned parametric curve (such as a DL model) to extrapolate outside of the training zone. But will this be meaningful, i.e. will it lead to generalization? Well, there are two possible settings.https://twitter.com/Gavin_Cawley/status/1450726111957209093 …

In practice, we encode priors in deep learning models in two ways: 1. Architecture patterns (e.g. convolution, recurrence, attention...) 2. Data augmentation (random-yet-valid input variations) They represent assumptions about the structure of your latent manifold.

Unfortunately, the kind of prior knowledge we inject in this way is very weak compared to the complexity and unpredictability of most real-world datasets. As a result, deep learning models cannot meaningfully extrapolate except on very simple problems (special cases).

Your DL-only domestic robot won't be able to make sense of a new kitchen layout. Your DL-only self driving car won't be able to train itself in Berlin then drive in London, etc.

Your model priors can't make that jump -- not even close. They can barely move you from "here's this scene" to "here's the same scene but a bit cloudy".

(Note: you could achieve this type of local, mini-extrapolation either via a brightness normalization step or via data augmentation, both of which encode prior knowledge about the characteristics of the visual world.)