The finding is very concise, and abstract sums it nicely.
Main work by Elad Ben Zaken, with contributions by @ravfogel https://twitter.com/yoavgo/status/1344769789328306176 …pic.twitter.com/PgWOqK5JF4
In general, only fine-tuning a small subset of the parameter on any large pretained model (in any domain) is competitive with / sometimes better than fine-tuning the entire model, because these models start overfitting quickly. It's a powerful form of regularization.
-
-
An interesting experiment in that context will be to train layers from the middle of the network and see if the same things happen.
-
some of the biases are in the middle of the network.
End of conversation
New conversation -
-
-
indeed. but we did not observe this to this extreme before in these kinds of models, and i am surprised that its the bias terms specifically that worked so well.
-
Are the biases special, or does the effect come from tuning a subset of the params of each layer?
- Show replies
New conversation -
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.