The finding is very concise, and abstract sums it nicely.
Main work by Elad Ben Zaken, with contributions by @ravfogel https://twitter.com/yoavgo/status/1344769789328306176 …pic.twitter.com/PgWOqK5JF4
-
indeed. but we did not observe this to this extreme before in these kinds of models, and i am surprised that its the bias terms specifically that worked so well.
Are the biases special, or does the effect come from tuning a subset of the params of each layer?
-
-
as you said, tuning more uniformly should be intuitively stronger. it does not seem to be the case. anyhow, for me the thing is not that the biases are special, but that they are sufficient.
Thanks. Twitter will use this to make your timeline better. UndoUndo
-
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.