Stupidly late to these comments but very well articulated, Blake.
-
-
Replying to @SussilloDavid @tyrell_turing and
I think it's perfectly reasonable for a scientist to choose to avoid a certain field of theoretical research because they don't believe the tools are there to falsify it experimentally.
1 reply 0 retweets 6 likes -
Replying to @pfau @SussilloDavid and
Yes, totally. But, that's precisely what we have to change (and what a few groups are working on, including mine): our credit assignment models need to start making physiological predictions that can be falsified.
2 replies 0 retweets 6 likes -
Replying to @tyrell_turing @pfau and
Wait, credit assignment = backprop?
2 replies 0 retweets 1 like -
Replying to @xaqlab @tyrell_turing and
The brain does credit assignment by whatever was cool at NeurIPS 5 years ago.
1 reply 0 retweets 3 likes -
Let’s add meat to this then, though. What are good options for credit assignment that *don’t* require efficient access to an estimate of the 1st order gradient of an objective function w.r.t. a given synaptic weight deep in a network? Honest question.
1 reply 1 retweet 3 likes -
Replying to @AdamMarblestone @pfau and
(I realize many answers will restrict the architecture or objective fxn greatly, to allow specialized non-backprop ways to get such gradients, which wouldn’t work for general fxn approx in arbitrary net topology. What can you really do with those? What’s the best alternative?)
2 replies 0 retweets 1 like -
Replying to @AdamMarblestone @pfau and
I guess it breaks into 2 cases: 1) you don’t need the gradient, or 2) you do but you have a way to get it that is structurally very different than backprop. For instance this paper gets gradient w/ either backprop or EM (perhaps an example of case #2): https://arxiv.org/abs/1202.3732
1 reply 0 retweets 3 likes -
Replying to @AdamMarblestone @xaqlab and
Also I think "the brain minimizes an objective function" is a vacuous statement. All dynamical systems can be reframed as solving a variational problem.
3 replies 0 retweets 5 likes -
Replying to @pfau @AdamMarblestone and
E.g. a flow field with curl is not the gradient of anything.
3 replies 0 retweets 4 likes
Right. The claim is not just that brain’s dynamics satisfies some arbitrary Euler-Lagrange equations, but that it’s learning dynamics actually corresponds to ongoing optimization during the lifetime of the organism, i.e., brain is itself carrying out an optimization algorithm .
-
-
Replying to @AdamMarblestone @xaqlab and
Ok. What about belief propagation? The fixed points are minima of the Bethe free energy, but it doesn't look much like gradient descent.
1 reply 0 retweets 0 likes -
I think a good example. So we have BP (backprop) and BP (belief propagation). Vicarious seems to like the latter. And people know to search for this in large scale cortical experiments, e.g., many of the IARPA MICRONS proposals centered on this in 2015 or so.
1 reply 0 retweets 0 likes - 11 more replies
New conversation -
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.