Show this thread
There's a need for an ambitious new AI benchmark that would measure the generalization power of a model -- the size of the radius in problem space around training data points for which the model can operate. Goal: maximize that radius while minimizing # of training data points.pic.twitter.com/EkIfpldyzd
-
-
-
Naturally, deep learning (smooth morphing from input to output) would not perform well on such a benchmark. Deep learning models require to be trained on a dense sampling of the data manifold -- they excel at pattern recognition, but cannot generalize far from the training data.
Show this thread -
The best kind of benchmark is one that sounds impossible at first -- like ImageNet back in the day. Having ambitious (but explicit) targets is what drives progress.
Show this thread
End of conversation
New conversation -
-
-
Isn’t the problem with this the same as kernel machines: how do you choose a meaningful measure of distance? Ideally your radius-extended blue set should resemble the test data distribution for this to be a measure of generalisation. What am I missing?
-
The point is to test on data that does not, in fact, resemble the training distribution. You are right that picking an appropriate definition for distance is important. This is domain-specific, there will not be one universal definition. But there can be one for images, or games
- Show replies
New conversation -
-
-
This Tweet is unavailable.
-
Quite. It's only possible to define a distance metric once the hypothesis space has been defined. So the key is to define very general hypothesis spaces and then have sparse models able to identify the important hypotheses from that space. We have done a fair bit of work on this.
End of conversation
-
-
-
Aren't SVMs basically the extreme of this?
Thanks. Twitter will use this to make your timeline better. UndoUndo
-
-
-
That's what OpenAI tried to work on with Universe, but the execution didn't quite pan out, due to tech issues but also because current mindsets still focus on datasets rather than data environments. Future research needs to stop requiring 'random' sampling.
Thanks. Twitter will use this to make your timeline better. UndoUndo
-
-
-
This sounds like this adversarial defense via convex optimization paper: https://arxiv.org/abs/1711.00851
Thanks. Twitter will use this to make your timeline better. UndoUndo
-
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.