Conversation

I’ve seen this happen: there’s a breakthrough, and people moving into the field in response can act on it faster than those already there, because the incumbents feel responsible to their now-obsolete research projects.

Quote Tweet

Michael Nielsen

@michael_nielsen

Jan 6

This happens often in research, where the most exciting research frontiers sometimes move faster than any individual researcher: twitter.com/michael_nielse

Show this thread

David Chapman

@Meaningness

Jan 6

The main obstacle to scientific progress is “accountability”: “you said you would do experiment X and we gave you $Y to do that, and you didn’t, you went off and did something unrelated. No grant renewal for you!!” “Yes but X is now pointless” falls on finger-filled ears.

David Chapman

@Meaningness

Jan 6

On the other hand, administrators have a responsibility to spend money wisely, and scientists have to be evaluated *somehow*. I don’t see any way out of this conundrum. It keeps me awake at 3am sometimes.

Replying to

I was referring to something slightly different: not how fast researchers can change direction in practice, but how fast they can do it even in principle.

Replying to

and

You're working on image recognition, you know your HOG & SIFT... & all of a sudden you're being decimated by young whipersnappers who know GPUs and CNNs. You can catch back up... but the field may well have moved again.

Replying to

and

I really mean literally: the frontier can move faster than any individual human is capable (well, apart maybe from a few like von Neumann...), not just than what administrators will allow. I think this is just a fascinating collective effect.

Replying to

and

Explicit model: + It may happen that the best expertise to work on some problem in 2022 is completely different than the best expertise needed in 2023 + It may require 5 years to master that new expertise + But there may already be pre-existing people with that expertise...

Replying to

Right… although maybe your example doesn’t support it? People didn’t move into vision with extensive gpu expertise (I think?) and also you can pick that up in a few weeks or months (I think?). Similarly probably for cnns when that took off (less confident on that?).

Replying to

My secondhand understanding is that Alex Krizhevsky had a lot of GPU expertise when AlexNet turned the image world upside down. A few years later and all that expertise was abstracted into libraries, but for a few years GPU expertise was an advantage.

Replying to

and

Arguably, GPU expertise is again relevant in some ML sub-fields. People want to do large experiments on academic budgets, or very large experiments on industrial lab budgets, and so will sometimes write CUDA kernels directly over using libraries for better performance.

11:15 PM · Jan 6, 2022Twitter Web App

Replying to

and

My favourite example is

@andy_l_jones

's paper "Scaling Scaling Laws with Board Games" (not a typo), where he studied scaling laws for RL (usually the domain of mega-labs) with a tiny grant budget, in part by implementing board games in CUDA. arxiv.org/pdf/2104.03113, Appendix A.