I am reaching a little bit but why not? @fchollet I am reading your paper: https://arxiv.org/abs/1911.01547 (for the 3rd time I think) why is it that generalization has necessarily a cost that makes it antagonistic to compression in the Occam's Razor sense? @RobertTLange?
-
Show this thread
-
Isn't there a benefit for the generalization capabilities of a system or skill program to be able to compress information? Reading now I have this intuition that generalization and compression have some kind of interplay but I might completely off...
1 reply 0 retweets 0 likesShow this thread -
Replying to @LucasEnkrateia
Compression (with respect to a fixed training dataset) is by definition opposite to generalization. If you find the optimal compression dict to compress English Wikipedia it will obviously not be optimal for Spanish Wikipedia.
2 replies 0 retweets 1 like -
Replying to @fchollet
First of all...how cool is it that I got a response from you...this is like so awesome, thanks a lot. With respect to the argument. Ok, I see your point on the dict to compress English, but wouldn't there be some compressed version of language that comprises both?
1 reply 0 retweets 0 likes -
Replying to @LucasEnkrateia
There might be, but it will be by definition less optimal when applied to just English. You won't find it by optimizing for English.
2 replies 0 retweets 1 like -
Replying to @fchollet @LucasEnkrateia
The more general point is that generalization requires you to store seemingly useless information at training time (info that doesn't help your training objective), that will become useful in the future (when generalization actually happens). That's the opposite of compression.
2 replies 0 retweets 1 like
Now, obviously, generalization requires abstraction, which requires erasing irrelevant details, so your high-generalization system will be doing *some* amount of compression. But it will be storing lots of seemingly useless info as well.
-
-
Replying to @fchollet
Is this why you argue in the paper that it is a necessity for a generalizable system to have a cost. Like it can't get away from storing useless info?
0 replies 0 retweets 0 likesThanks. Twitter will use this to make your timeline better. UndoUndo
-
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.