The race in AI should be toward models with fewer and fewer parameters for a given performance, not the opposite.
-
It's "a few" because (1) only "a few" have easy access to the required computing resources. (2) distributing large models on multiple GPUs and training them on many multi-GPU nodes requires a lot of infrastructure engineering. That's only available in a few industry labs.
You’re confusing the head of the power law with the whole distribution. Everyone and their cat is trying to learn larger and larger models to the extent they can. Also, by Moore’s law etc. the requirements keep getting easier to meet.
