I meet lots of people who tell me fatalistically (& often despondently) that it's near impossible to do important work on neural nets today, unless you have huge compute and huge data sets.
Show this thread
-
In science, the most important progress often comes from better questions and better ideas, not better equipment (in this case, more computational power and data).
Show this thread
In biology, the government-run genome project cost 10 times as much as Venter's private project. Much of the reason for the cost difference is that Venter adopted a clever hack (pairwise end shotgun sequencing) the government project didn't use until the end.
-
-
In particle physics, Freeman Dyson found that, contrary to conventional wisdom, only a small fraction of the most important progress comes from building bigger accelerators. Much of it comes from much harder-to-control improvements in detectors and the like.pic.twitter.com/J2ejGimM0t
Show this thread -
Why is big science so seductive? In part, because it seems guaranteed: you can plan, you can see success from the start. That's much less nerve-inducing (and _seems_ less uncertain) than needing to have clever ideas along the way.
Show this thread -
Back to neural nets: a danger in scaling up your computational power is that you start to focus _only_ on questions that require that computational power. You hire specialists who thrive in that environment, but who aren't so good at playing with basic, fundamental questions...
Show this thread -
... and your culture starts to tilt that way, driving out people who do like to play with basic, fundamental questions.
Show this thread -
Take all this with a grain of salt. Neural nets are a side interest, not my main interest. Maybe I'm wrong. But I don't think so. This dynamic has played out in genome sequencing, in particle physics, & in many other areas. Big science is attractive, but often small science wins
Show this thread -
Does this mean computational power or big data is useless? No, of course not. There are important questions that can likely only be addressed that way. But if you want to work on AI, it seems to me a mistake to be too focused on the need for lots of data and lots of compute.
Show this thread
End of conversation
New conversation -
-
-
I'm well aware. That's why I wrote "much of the reason".
- 1 more reply
New conversation -
-
Venter’ s strategy was clever, but much of cost of HGP went into developing the tech Venter used and developing mapping info Celera didn’t.
Thanks. Twitter will use this to make your timeline better. UndoUndo
-
-
-
true that. However, many young scientist don't get to try their clever ideas without embedding in a lab with (big) machines...
Thanks. Twitter will use this to make your timeline better. UndoUndo
-
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.