Since many of the interesting machine learning papers now regularly required 100s or even 1000s of CPU/GPUs for replication - what strategies are realistically left for startups, public institutions & individuals to do meaningful research in ML?
-
A lot of important work comes from the constraint of having small compute. In ~2011 Ng et al attacked ImageNet with 1000+ CPUs. In 2012, Hinton et al did way better with 2 GPUs. In general, less compute often means people need better ideas. Which isn't a tragedy!
When I think of my own favourite papers, many of them are on single machines. It was a good idea or good question, not having a giant cluster.
-
-
Plus, work that develops fundamental ideas (think dropout, batch norm, CapsNets,..) often starts on tractable datasets (MNIST, CIFARs, ..). ImageNet-scale research is often done by people/companies with enough resources and almost always doesn't attack the fundamentals.
Thanks. Twitter will use this to make your timeline better. UndoUndo
-
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.