Question: Is there a way to not break the orthogonality of weight matrices by backprop, given your n/w was initialized with othogonal Ws?
In 2015, I've tried doing so by adding a loss that elastically encourages orthogonality. Results weren't clearly positive.
-
-
By elastically you mean using an elastic net? I might be misunderstanding. If not, what do you mean? I am curious.
-
I just mean that since it's a loss, it doesn't strictly enforces orthogonality. If breaking orthogonality is useful, the model will break it
- Show replies
New conversation -
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.