François CholletVerified account

Usual reminder: when I've been saying for the past 5+ years that deep learning is interpolative, I don't mean it does linear interpolation in the original encoding space (which would be useless). It does interpolation on a low-dimensional manifold embedded in the encoding space.

I’ve wondered a bit / gone down rabbit holes in the recent past about the connections between language embeddings and topology, but it would be fascinating to see if there are task / data independent forms that emerge in the low-dim manifold where interpolation takes place.