I was referring to something slightly different: not how fast researchers can change direction in practice, but how fast they can do it even in principle.
Conversation
You're working on image recognition, you know your HOG & SIFT... & all of a sudden you're being decimated by young whipersnappers who know GPUs and CNNs. You can catch back up... but the field may well have moved again.
1
3
I really mean literally: the frontier can move faster than any individual human is capable (well, apart maybe from a few like von Neumann...), not just than what administrators will allow.
I think this is just a fascinating collective effect.
1
3
Explicit model:
+ It may happen that the best expertise to work on some problem in 2022 is completely different than the best expertise needed in 2023
+ It may require 5 years to master that new expertise
+ But there may already be pre-existing people with that expertise...
2
8
Replying to
Right… although maybe your example doesn’t support it? People didn’t move into vision with extensive gpu expertise (I think?) and also you can pick that up in a few weeks or months (I think?). Similarly probably for cnns when that took off (less confident on that?).
2
2
Replying to
My secondhand understanding is that Alex Krizhevsky had a lot of GPU expertise when AlexNet turned the image world upside down. A few years later and all that expertise was abstracted into libraries, but for a few years GPU expertise was an advantage.
2
2
Arguably, GPU expertise is again relevant in some ML sub-fields.
People want to do large experiments on academic budgets, or very large experiments on industrial lab budgets, and so will sometimes write CUDA kernels directly over using libraries for better performance.
1
2
My favourite example is 's paper "Scaling Scaling Laws with Board Games" (not a typo), where he studied scaling laws for RL (usually the domain of mega-labs) with a tiny grant budget, in part by implementing board games in CUDA.
arxiv.org/pdf/2104.03113, Appendix A.
2
3
(And not just board games, I should add: also a full implementation of Monte Carlo tree search, which is normally run on a CPU!)
1
1
Thanks Nicholas! Both for the compliment and for tagging me in an interesting thread
2
2
Glad you didn't mind the tag :-)
I'm never sure whether writing out people's names or tagging their Twitter handles is more polite.