The main obstacle to scientific progress is “accountability”: “you said you would do experiment X and we gave you $Y to do that, and you didn’t, you went off and did something unrelated. No grant renewal for you!!”
“Yes but X is now pointless” falls on finger-filled ears.
Conversation
On the other hand, administrators have a responsibility to spend money wisely, and scientists have to be evaluated *somehow*.
I don’t see any way out of this conundrum.
It keeps me awake at 3am sometimes.
6
1
21
Replying to
I was referring to something slightly different: not how fast researchers can change direction in practice, but how fast they can do it even in principle.
1
2
You're working on image recognition, you know your HOG & SIFT... & all of a sudden you're being decimated by young whipersnappers who know GPUs and CNNs. You can catch back up... but the field may well have moved again.
1
3
I really mean literally: the frontier can move faster than any individual human is capable (well, apart maybe from a few like von Neumann...), not just than what administrators will allow.
I think this is just a fascinating collective effect.
1
3
Explicit model:
+ It may happen that the best expertise to work on some problem in 2022 is completely different than the best expertise needed in 2023
+ It may require 5 years to master that new expertise
+ But there may already be pre-existing people with that expertise...
2
8
Replying to
Right… although maybe your example doesn’t support it? People didn’t move into vision with extensive gpu expertise (I think?) and also you can pick that up in a few weeks or months (I think?). Similarly probably for cnns when that took off (less confident on that?).
2
2
Replying to
My secondhand understanding is that Alex Krizhevsky had a lot of GPU expertise when AlexNet turned the image world upside down. A few years later and all that expertise was abstracted into libraries, but for a few years GPU expertise was an advantage.
2
2
Arguably, GPU expertise is again relevant in some ML sub-fields.
People want to do large experiments on academic budgets, or very large experiments on industrial lab budgets, and so will sometimes write CUDA kernels directly over using libraries for better performance.
1
2
My favourite example is 's paper "Scaling Scaling Laws with Board Games" (not a typo), where he studied scaling laws for RL (usually the domain of mega-labs) with a tiny grant budget, in part by implementing board games in CUDA.
arxiv.org/pdf/2104.03113, Appendix A.
2
3
(And not just board games, I should add: also a full implementation of Monte Carlo tree search, which is normally run on a CPU!)
Thanks Nicholas! Both for the compliment and for tagging me in an interesting thread
2
2
I agree where the frontier is can shift faster than any individual, but I think the gap between 'frontier speed' and 'max researcher speed' is less important than the gap between 'max researcher speed' and 'average researcher speed'. If only for growth-mindset reasons!
1
3
Show replies