A randomly weighted Wide ResNet-50 contains a subnetwork that is smaller than, but matches the performance of ResNet-34 on ImageNet :o (2/n)
Prikaži ovu nit
What's hidden in an overparameterized neural network with random weights? If the distribution is properly scaled (e.g. Kaiming Normal), then it contains a subnetwork which achieves high accuracy without ever modifying the values of the weights... https://arxiv.org/abs/1911.13299 (/n)pic.twitter.com/RcTcgYGY9J
-
-
-
On CIFAR-10, the performance of a randomly weighted subnetwork matches that of the learned dense model as the width (e.g. number of channels) of the network increases (3/n)pic.twitter.com/pvp7zCjBSr
Prikaži ovu nit -
Joint work with
@RamanujanVivek who is applying to grad school (!!), and other amazing coauthors@anikembhavi, Ali Farhadi, and@morastegari. (4/n)Prikaži ovu nit -
Alternate title: Randomly weighted neural networks. What do they contain? Do they contain things? Lets find out. https://arxiv.org/abs/1911.13299
Prikaži ovu nit -
This work is inspired by and builds upon the incredible foundational work of
@oh_that_hat et al. in Deconstructing the LTH, Gaier and@hardmaru in Weight Agnostic Neural Networks,@jefrankle and@mcarbin in the LTH, and many many more -- check out these amazing/inspiring papers!Prikaži ovu nit
Kraj razgovora
Novi razgovor -
-
-
Great paper! I appreciate the PyTorch snippet at the end!
-
Thank you we are working on a full code release hopefully ready real soon
- Još 1 odgovor
Novi razgovor -
-
-
Interesting. Without going into the details of the paper, what is the high-level killer argument for the existence of such a subnet ?
-
Great q, here is some intuition i found useful: The number of possible subnetworks is unthinkably big! It’s combinatorial in the size of the network, and modern networks have millions of params. As such there should be a subnetwork that performs well.
Kraj razgovora
Novi razgovor -
-
-
We recently found that a randomly initialized + fine-tuned BERT performs surprisingly well in 5/6 NLP tasks (80% acc for sentiment analysis!). I guess fine-tuning could be interpreted as tweaking the net so as to amplify the successful subnetwork? Paper: https://www.aclweb.org/anthology/D19-1445.pdf …pic.twitter.com/5fFvFq1bVL
-
Whoa very cool! (
@gabriel_ilharco check this out ^) - Još 1 odgovor
Novi razgovor -
Čini se da učitavanje traje već neko vrijeme.
Twitter je možda preopterećen ili ima kratkotrajnih poteškoća u radu. Pokušajte ponovno ili potražite dodatne informacije u odjeljku Status Twittera.