The finding is very concise, and abstract sums it nicely.
Main work by Elad Ben Zaken, with contributions by @ravfogel https://twitter.com/yoavgo/status/1344769789328306176 …pic.twitter.com/PgWOqK5JF4
-
An interesting experiment in that context will be to train layers from the middle of the network and see if the same things happen.
It should be more effective to distribute the trainable parameters throughout the model rather than just train one single layer of equivalent size somewhere in the model. Experiment would be to train random subset of each weight variable (can be implemented with projections)
-
-
we compare to previous method (diff-pruning) that learns additive changes on top of the entire network, with sparse regularization. the simple bias method "wins", mostly.
Thanks. Twitter will use this to make your timeline better. UndoUndo
-
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.