Pedro Domingos

The race in AI should be toward models with fewer and fewer parameters for a given performance, not the opposite.

While there are a few scientists training impractically large models, there are infinitely more engineers using distillation, pruning, quantization, and architecture optimization to get the same result with smaller and deployable models.

It’s more than a few scientists. It’s a major trend in AI, spearheaded by large groups at several top AI labs.

It’s not a good trend. AI research becomes a monopoly game for people who can have access to 1,000s of GPUs and data storage that can hold the Common Crawl…