Not all functions we want to learn can be built efficiently from differentiable parts. What's less obvious to me right now: could all the important metalearning functions turn out to be differentiable? (not that they have to!)
-
-
Isn't the main problem the presence of large & many discontinuities (~by definition of differentiability). Looping computations are frequently fairly smooth (each iteration brings you a bit closer to the result). Branching (if-statements) OTOH cause very sharp discontinuities.
-
It is very easy to entrain a neural network with function that performs an if statement, and hard to entrain it with a loop.
- 2 more replies
New conversation -
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.