François CholletVerified account

In such a case your learned manifold will feature lots of areas that aren't actually meaningful as per the original domain, but that could be interpreted as a continuous extension of the domain (according to the model). Like reals are to integers.

This enables you to solve seemingly-discrete problems with deep learning -- as long as they are so structured as to be interpolative, and as long as you can gather a dense sampling of the manifold. The manifold hypothesis must apply.

However, when faced with a problem where the manifold hypothesis no longer applies, fitting a differentiable curve via gradient is no longer a good strategy. This is actually *most* of the space of all interesting/useful problems! E.g. most programs that software engineers write.

Ah, but this is incorrect! Virtually all perception problems *do* feature entirely-connected latent manifolds. Natural images are no exception. This is very easy to demonstrate, and something that cinematographers are intimately familiar with...https://twitter.com/david_picard/status/1450697372930088963 …

Take a picture of your dog in your backyard, and a picture of your coffee mug on your kitchen table. Unconnected, right? You can't morph one into the other in a continuous fashion, right? Of course you can!

There's an infinity of ways you could do it, but here's a few trivial ones.

1. The handheld camera: just keep the camera rolling as you walk from your kitchen to your backward. Boom, a smooth transition where *every single frame* is extracted from the real world.