We won’t find intelligence at the bottom of gradient descent.
-
Local or not local, swarm or not swarm, the things you mention are just particular ways to do gradient-free optimization.
To paraphrase Ulam, using a term like "gradient-free optimization" is like referring to the bulk of zoology as the study of trunk-free animals. GD is fast per step, but extremely inefficient at extracting information from examples. (Nearest-neighbor is better at that!)
-
-
What you're doing is like arguing a billion years ago that bacteria were the apex of evolution. (They search for food by concentration gradient descent - do you have a better option?) You'll doubtless be proved right in the short term, but (very) wrong in the long term.
-
Even GD's speed advantage is predicated on the unfair advantage of GPUs, and less than it seems given how overparameterized models have to be to overcome local optima - which nonconvex optimizers don't. (E.g., https://homes.cs.washington.edu/~pedrod/papers/iclr18.pdf …)
Keskustelun loppu
Uusi keskustelu -
-
-
You can measure how well nearest-neighbor "extracts information from examples" by counting how many samples it needs to reach a given error level. Do the same with a neural net trained with gradient descent. The result will not support your statement.
-
If you use the dumbest possible form of nearest-neighbor and the smartest possible form of gradient descent, yes. If you're smart about how you map test examples to training ones, no. In any case, I was talking about the information extracted per step, where kNN blows GD away.
- Näytä vastaukset
Uusi keskustelu -
Lataaminen näyttää kestävän hetken.
Twitter saattaa olla ruuhkautunut tai ongelma on muuten hetkellinen. Yritä uudelleen tai käy Twitterin tilasivulla saadaksesi lisätietoja.