Further experimentation done by the @fastdotai community on how to combine the new optimizers that have just come out. This is why I love being a part of it. Sota results on limited epoch Imagewoof training which correlates directly with Imagenet performance. Great work!https://twitter.com/federicolois/status/1166919084052754432 …
Also note they added self-attention and mish (relu replacement) to the mix. Can’t wait to dig into this more tonight. Self-attention in particular to me seems like it really should be given a lot more...ahem...attention.
-
-
I've got an architecture that relies on self attention that I've been wanting to try for a while. It's such an interesting method.
-
Yeah it’s hugely powerful in my experience. The only thing you have to be careful about is numerical stability
End of conversation
New conversation -
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.