Interesting new "Simple Self Attention layer" that improves all tested resnet baselines on Imagenette and Imagewoof. Seems like something worth looking into further! https://github.com/sdoria/SimpleSelfAttention …pic.twitter.com/PIIy0nXHfm
-
I've made an injection of Simple Self-Attention layer to a pretrained Resnet. It could be more practical to use it that way. It works good, although it is hard to tell is it really better than plain Resnet. https://github.com/mgrankin/SimpleSelfAttention/blob/master/pretrained.py …pic.twitter.com/KibuEamHMP
Have you been tracking gamma? You'll want to train until that levels off a bit. Self-attention tends to take a while to train. I'm also curious how this compares to PooledSelfAttention (in FastAI library). I've been using that lately and it's next level for me.
-
-
BTW I should have mentioned- great idea doing this injection!
Thanks. Twitter will use this to make your timeline better. UndoUndo
-
-
-
Which gamma?
-
In the self-attention forward: o = self.gamma * o + x . i.e. the weighting of attention.
- Show replies
New conversation -
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.