François CholletVerified account

Isn’t the problem with this the same as kernel machines: how do you choose a meaningful measure of distance? Ideally your radius-extended blue set should resemble the test data distribution for this to be a measure of generalisation. What am I missing?

The point is to test on data that does not, in fact, resemble the training distribution. You are right that picking an appropriate definition for distance is important. This is domain-specific, there will not be one universal definition. But there can be one for images, or games

So you’re testing a special case of domain adaptation. Presumably, if you know the definition of the distance you could just train with corresponding data augmentation? Say, for Euclidean distance use spherically symmetric additive noise.

I think you're hopelessly missing the point.

It was a pleasure engaging in a discussion about your idea. I’m sorry you had to give up on me.

In case you were actually expecting a reply: the purpose of this property is that it is useful in the real world. You might want, for example, to train a robot to operate in a couple kitchen then have it operate in an unknown kitchen. Which humans can do.

A DL model would have to be trained on a dense sampling of all possible kitchens. In the real world it's easy to come up with novel test data points but impossible to have a formal definition for generalization distance, let alone a way to use data aug. to cover the full space.

Naturally you know this and you weren't expecting me to point it out. You frequently send me these bad faith adversarial/ironic replies where you play dumb to make a point. Too used to it to "engage", sorry. It was my mistake for even replying in the first place