A common beginner mistake is to misunderstand the meaning of the term "interpolation" in machine learning.
Let's take a look 
Show this thread
-
Your overall point is that "interpolation" requires specifying a representation (manifold + distance) over which the computation is performed. One CAN interpolate in the image space, and one can interpolate in a fitted manifold. The latter works much better, of course.
The former only leads to generalization in the case where the data manifold is Euclidean (e.g. linear regression)
