michael_nielsen

If you are reeling from a NeurIPS rejection or stressing about an ICLR submission, remember that some of the best papers were never published anywhere except arxiv. Thread of a few favorites (1/5):

"Generating Sequences with RNNs" by Graves https://arxiv.org/abs/1308.0850  This paper blew my mind when it came out, showing that it was possible to generate plausible text and handwriting with RNNs. Includes the predecessors of attention, Adam, etc... (2/5)

WaveNet by van den Oord et al. https://arxiv.org/abs/1609.03499  Until this came out I don't think most of us expected that we'd be able to generate raw waveforms with deep networks anytime soon. The results were surprisingly good and the architecture remains influential. (3/5)

"Learning to Generate Reviews and Discovering Sentiment" by Radford et al. https://arxiv.org/abs/1704.01444  A simple and surprising result (thresholding a neuron in an unsupervised LM could classify sentiment accurately) that helped kicked off the transfer learning craze in NLP. (4/5)

"Implicit Autoencoders" by Makhzani https://arxiv.org/abs/1805.09804  A deeper cut, this paper (loosely speaking) proposes a VAE where both the "reconstruction" and "regularization" terms are replaced with adversarial losses. Impressive results on disentangling style and content. (5/5)