Since many of the interesting machine learning papers now regularly required 100s or even 1000s of CPU/GPUs for replication - what strategies are realistically left for startups, public institutions & individuals to do meaningful research in ML?
-
excellent points, thanks!
A lot of important work comes from the constraint of having small compute. In ~2011 Ng et al attacked ImageNet with 1000+ CPUs. In 2012, Hinton et al did way better with 2 GPUs. In general, less compute often means people need better ideas. Which isn't a tragedy!
-
-
The question was provoked by my own at times out of control EC2 costs. Even doing small scale "creative" research does get rather expensive quickly. But i do totally agree with the sentiment of what your saying & will try to be more creative ;)
-
GPU on aws is really poor value, because they pay the nvidia server tax. Try renting gtx 1060 on ovh
- 2 more replies
New conversation -
-
-
In addition to this excellent point, I'd add that IMO many recent papers using big compute, whilst getting a lot of PR, were not actually interesting or useful. I've noticed that many researchers with access to lots of compute seem to have completely wrong idea of what's useful
-
For pure research 100% agree. In applied work, i had many cases where compute was a primary challenge. e.g. recently: client wants to use a WaveNet-like architecture to generate many voices. Requires lot's of experimentation but blows budget v.quick. Or using HD-GAN's, etc.
- 3 more replies
New conversation -
-
-
When I think of my own favourite papers, many of them are on single machines. It was a good idea or good question, not having a giant cluster.
-
Plus, work that develops fundamental ideas (think dropout, batch norm, CapsNets,..) often starts on tractable datasets (MNIST, CIFARs, ..). ImageNet-scale research is often done by people/companies with enough resources and almost always doesn't attack the fundamentals.
End of conversation
New conversation -
-
-
Agreed, that trend continues today. Many default "train in afternoon" framework examples would've hit SotA a year or two ago. Smarter techniques >> more resources. Hence imho you can make meaningful / SotA contributions with the standard processing power of today.
Thanks. Twitter will use this to make your timeline better. UndoUndo
-
-
-
Back in early 90’s when I was a researcher in distributing FEM I met a bunch of Soviet refugees. Best programmers. As they were constrained by compute power restricted to their versions of the PC. When they got our power they were unstoppable. Constraints can be good.
Thanks. Twitter will use this to make your timeline better. UndoUndo
-
-
-
now imagine how innovative you have to be to do quantum machine learning with <50 qubits...
Thanks. Twitter will use this to make your timeline better. UndoUndo
-
-
-
Why reference Hinton et al and not Krizhevsky et al for the 2012 work? Hinton didn’t write cuda-convnet, after all…
Thanks. Twitter will use this to make your timeline better. UndoUndo
-
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.