One interesting thing about deep learning is that even as ever better results surface, everything we know about NNs is probably wrong. A short list (in rough chronological order): - "you need to pretrain a NN" - "NNs require thousands of datapoints to train" (1/10)
-
Show this thread
-
- "NNs must be trained by backpropagation" - "hybrid approaches like SVMs on top of NN features will always work better" - "backpropagation in any form is biologically implausible" - "CNNs are nothing like the human visual cortex & certainly don't predict its activations"
2 replies 0 retweets 12 likesShow this thread -
- "small NNs can't be trained directly, so NNs must need to be big" - "NNs only learn task-specific features, certainly no kind of hidden or latent 'dark knowledge'" - [style transfer arrives] "Who ordered that?" - "simple SGD is the worst update rule"
1 reply 0 retweets 11 likesShow this thread -
- "simple SGD is the worst update rule" - "simple self-supervision like next-frame prediction can't learn semantics" - "adversarial examples're easy to fix & won't transfer, well, won't blackbox transfer, well, won't transfer to realworld, well..." - [batchnorm arrives] "Oops."
1 reply 0 retweets 10 likesShow this thread -
- "big NNs overfit by memorizing data" - "you can't train 1000-layer NNs but that's OK, that wouldn't be useful anyway" - "big minibatches don't generalize" - "NNs aren't Bayesian at all"
1 reply 0 retweets 10 likesShow this thread -
- "convolutions are only good for images; only LSTM RNNs can do translation/seq2seq/generation/meta-learning" - "you need small learning rates, not superhigh ones, to get fast training" (superconvergence) - "memory/discrete choices aren't differentiable"
1 reply 0 retweets 8 likesShow this thread -
- [CycleGAN arrives] "Who ordered that?" - "you can't learn to generate raw audio, it's too low-level" - "you need bilingual corpuses to learn translation" - "you need shortcut connections, not new activations or initializations to train 1000-layer nets"
2 replies 1 retweet 10 likesShow this thread -
Replying to @gwern
How do you learn translation without a bilingual corpus? Seems impossible, a priori; I must be missing something. Citation?
2 replies 0 retweets 2 likes -
Replying to @Meaningness @gwern
Unless this is the language A->C->B without an A/C corpus thing? Which is kinda function composition.
1 reply 0 retweets 1 like
I mean without A/B corpus
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.