I really mean literally: the frontier can move faster than any individual human is capable (well, apart maybe from a few like von Neumann...), not just than what administrators will allow.
I think this is just a fascinating collective effect.
Conversation
Explicit model:
+ It may happen that the best expertise to work on some problem in 2022 is completely different than the best expertise needed in 2023
+ It may require 5 years to master that new expertise
+ But there may already be pre-existing people with that expertise...
2
8
Replying to
Right… although maybe your example doesn’t support it? People didn’t move into vision with extensive gpu expertise (I think?) and also you can pick that up in a few weeks or months (I think?). Similarly probably for cnns when that took off (less confident on that?).
2
2
Replying to
My secondhand understanding is that Alex Krizhevsky had a lot of GPU expertise when AlexNet turned the image world upside down. A few years later and all that expertise was abstracted into libraries, but for a few years GPU expertise was an advantage.
2
2
Arguably, GPU expertise is again relevant in some ML sub-fields.
People want to do large experiments on academic budgets, or very large experiments on industrial lab budgets, and so will sometimes write CUDA kernels directly over using libraries for better performance.
1
2
My favourite example is 's paper "Scaling Scaling Laws with Board Games" (not a typo), where he studied scaling laws for RL (usually the domain of mega-labs) with a tiny grant budget, in part by implementing board games in CUDA.
arxiv.org/pdf/2104.03113, Appendix A.
2
3
(And not just board games, I should add: also a full implementation of Monte Carlo tree search, which is normally run on a CPU!)
1
1
Thanks Nicholas! Both for the compliment and for tagging me in an interesting thread
2
2
I agree where the frontier is can shift faster than any individual, but I think the gap between 'frontier speed' and 'max researcher speed' is less important than the gap between 'max researcher speed' and 'average researcher speed'. If only for growth-mindset reasons!
1
3
I also think average researcher speed tends to fall short for compelling-if-disappointing reasons: that most researchers coevolve into a comfortable, pays-the-bills niche and it takes a certain kind of insanity to dump that all to go be awful and unpaid at something new + risky
2
3
I'm not sure if "pays-the-bills" was meant to be literal or metaphorical there, but I've found that working on something really different is often hard from a literal-bill-paying perspective.
In part because of things like this:
Quote Tweet
A colleague, in a department that shall not be named, received the following question from a grant admin:
“Can you please confirm that the mousepad was used solely for project X? If not, please transfer this expense off the project”
Show this thread
And in part because existing research infrastructure (grants, students, etc.) often turns into a commitment: if you promised a funding agency something it can be hard to go work on something else and hope to be funded in the future.
1