Question: Is there a way to not break the orthogonality of weight matrices by backprop, given your n/w was initialized with othogonal Ws?
-
By elastically you mean using an elastic net? I might be misunderstanding. If not, what do you mean? I am curious.
I just mean that since it's a loss, it doesn't strictly enforces orthogonality. If breaking orthogonality is useful, the model will break it
-
-
I see. How did you enforce orthogonality?
-
loss = sum of inner product of weights times scaling coefficient.
- Show replies
New conversation -
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.