These discussions relating deep networks and the brain often feel very cargo cultish, along with studies that apply neuroscience methods to deep networks(?).
-
-
Tell me more. What is the field trying to emulate, cargo cult style
2 replies 0 retweets 0 likes -
E.g. trying to explicitly implement backprop with spike-based learning rules. Gradient descent is ok-ish as an optimisation approach, but is there any reason to believe that bio-neural loss functions and optimisation approaches work like backprop in FFwd ANNs?
1 reply 0 retweets 0 likes -
It's not just okayish, it's the best we have for high D systems...
1 reply 0 retweets 0 likes -
Yeah, but it’s not very good at getting away from initial conditions is it?
1 reply 0 retweets 0 likes -
What are you basing that claim on?
1 reply 0 retweets 0 likes -
Replying to @tyrell_turing @DylanRMuir and
Cause to be clear, it's false. Gradient descent *depends* on initial conditions, but it can easily go far away from them.
4 replies 0 retweets 1 like -
Anyway, discussing exactly how good or bad gradient descent is, was not exactly the point of the conversation.
1 reply 0 retweets 1 like -
Replying to @DylanRMuir @tyrell_turing and
Brain could both do something broadly backprop-like for weight updating AND have interesting, structured initial architecture, e.g., cortical columns, sparsity, hierarchy... of course... to study one is not to exclude the other...
1 reply 0 retweets 3 likes -
Replying to @AdamMarblestone @tyrell_turing and
Yeah absolutely. Brains most likely do architecture search in evolutionary time; do a lot of structure initialisation on developmental time; and then do some local or semi-local learning to tune things up. But the boundaries between are not yet clear on the Neuroscience side.
1 reply 0 retweets 1 like
Yeah, I'd definitely agree with the first two, and then would add "and/or MAYBE deep..." to your "local or semi-local to tune things up"...
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.