Minimizing a cost function using SGD/backprop is like physics in much the same way as eigenvectors are part of quantum mechanics.
-
-
It occurs to me that I don't know where the theory of mathematical optimization (& notions like objective function, gradient descent) comes from. Certainly, it was buzzing as a subject independent of physics by the 1950s. But its roots may be in part in physics, earlier.
5 replies 0 retweets 7 likes -
Replying to @michael_nielsen @IntuitMachine
One major strand in the history would be cybernetics/control theory, which was applied physics initially—the radar-controlled antiaircraft gun being the paradigm for the field
2 replies 0 retweets 6 likes -
Replying to @Meaningness @michael_nielsen
This is the long forgotten connection. What happened was that the Dartmouth AI folk didn't invite Wiener because his approach was very different from emerging computer paradigm.
2 replies 0 retweets 0 likes -
Replying to @IntuitMachine @michael_nielsen
Right. And that antipathy led to the Minsky/Papert Perceptrons book, and then the standard histrionics takes over. But the mid-80s backprop people were very big on physics analogies, as I recall it, and that shaped the texture of the field.
2 replies 0 retweets 1 like -
Replying to @Meaningness @michael_nielsen
True. Hopfield was a physicist.
1 reply 0 retweets 0 likes -
Replying to @IntuitMachine @michael_nielsen
And the Boltzman machine was a big deal for about a year then.
1 reply 0 retweets 0 likes -
Replying to @Meaningness @michael_nielsen
Yes, without a doubt these are physics inspired models. They kicked started the deep learning field. But, few use these models anymore.
1 reply 0 retweets 0 likes -
Replying to @IntuitMachine @michael_nielsen
Yes; I understand and applaud your frustration that potentially relevant physical methods and intuitions are now being ignored.
1 reply 0 retweets 0 likes -
Replying to @Meaningness @michael_nielsen
Honestly, I am not sure if it's relevant anymore. We've entered the realm of massive computation and likely biology is more important than physics.
2 replies 0 retweets 0 likes
As in, neuroscience is a better source of analogies?
-
-
Replying to @Meaningness @michael_nielsen
Are they any closer in figuring out how to perform analogies?
1 reply 0 retweets 0 likes -
Replying to @IntuitMachine @michael_nielsen
Uh… I wasn’t sure what you meant by “biology is more important than physics,” and this was a guess
1 reply 0 retweets 0 likes - 1 more reply
New conversation -
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.