michael_nielsen

I meet lots of people who tell me fatalistically (& often despondently) that it's near impossible to do important work on neural nets today, unless you have huge compute and huge data sets.

It seems to me this is based on a bad model of how scientific progress gets made.

You can see this in part by looking at historically important discoveries: what do backpropagation, conv nets, Alexnet, GANS, LSTMs, ReLUs all have in common? All were developed with relatively small compute, and small data sets.

In science, the most important progress often comes from better questions and better ideas, not better equipment (in this case, more computational power and data).

In biology, the government-run genome project cost 10 times as much as Venter's private project. Much of the reason for the cost difference is that Venter adopted a clever hack (pairwise end shotgun sequencing) the government project didn't use until the end.

Why is big science so seductive? In part, because it seems guaranteed: you can plan, you can see success from the start. That's much less nerve-inducing (and _seems_ less uncertain) than needing to have clever ideas along the way.

Back to neural nets: a danger in scaling up your computational power is that you start to focus _only_ on questions that require that computational power. You hire specialists who thrive in that environment, but who aren't so good at playing with basic, fundamental questions...