François CholletVerified account

Two recent algorithms, World Models by @hardmaru and Schmidhuber (2018) and Curiosity by @pathak2206 et al. (2018), have approx. equal performance when a learned module (RNN and embedding in ICM, respectively) is instead left as a fixed randomly initialized module. @_brohrer_

Random weights have often been a surprisingly strong baseline. Back in 2007-2011 we often saw convolutional nets with random weights getting state of the art performance on object recognition tasks like CalTech-101 and CIFAR-10

Feature extraction via projection on a large family of random filters. Works especially well if your weights have an orthogonal initialization (so you don't lose to much information in the process).

I was going to add a tweet but it didn't go through. What's interesting here is not that a random network would work, it's that training this random network does not improve its performance. It's a sign there's a fundamental flaw with how we think about the setup.