Two recent algorithms, World Models by @hardmaru and Schmidhuber (2018) and Curiosity by @pathak2206 et al. (2018), have approx. equal performance when a learned module (RNN and embedding in ICM, respectively) is instead left as a fixed randomly initialized module.
@_brohrer_
-
Show this thread
-
Replying to @LiamFedus @hardmaru and
Random weights have often been a surprisingly strong baseline. Back in 2007-2011 we often saw convolutional nets with random weights getting state of the art performance on object recognition tasks like CalTech-101 and CIFAR-10
11 replies 25 retweets 160 likes -
Replying to @goodfellow_ian @LiamFedus and
Feature extraction via projection on a large family of random filters. Works especially well if your weights have an orthogonal initialization (so you don't lose to much information in the process).
3 replies 2 retweets 43 likes -
Replying to @fchollet @goodfellow_ian and
Yes, thanks! Another friend also referred us to the Johnson-Lindenstrauss lemma in random projections. Would your expectation be that projection on a large family of random filters would work as well as a random fixed CNN, both with the same output dimension?
1 reply 0 retweets 8 likes
I'd expect the random CNN to work better. Would be an interesting experiment.
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.