Self-attention is proving to be a really good unified prior for both image and sequence processing. It can prob also learn useful representations for images that are difficult for conv layers to learn. See author’s thread for blog post and web demo: https://twitter.com/jb_cordonnier/status/1215581826187743232?s=21 …https://twitter.com/jb_cordonnier/status/1215581826187743232 …
-
-
Prikaži ovu nit -
With recent work that enables transformer to process very long training sequences, we could be only scratching the surface of the full capabilities of self-attention networks. They may have strong inductive bias to model things like hi-res video sequences. https://twitter.com/hardmaru/status/1210912823221440514?s=21 …https://twitter.com/hardmaru/status/1210912823221440514 …
Prikaži ovu nit
Kraj razgovora
Novi razgovor -
-
-
Hvala. Twitter će to iskoristiti za poboljšanje vaše vremenske crte. PoništiPoništi
-
-
-
I predicted that self-attention was learning something similar to convolutions after reading https://twitter.com/quocleix/status/1120498893832237057 … - one more feature engineering bites the dust of the bitter lesson?
Hvala. Twitter će to iskoristiti za poboljšanje vaše vremenske crte. PoništiPoništi
-
-
-
Both are basically dot products. Multi-head attention is even more like convolutions.
Hvala. Twitter će to iskoristiti za poboljšanje vaše vremenske crte. PoništiPoništi
-
-
-
Something related: https://arxiv.org/abs/1905.01289
-
Yes, very nice work by J.-M. Andreoli (2019). Check out our paper to see how we relate to his work. Convolution, attention and structure embedding Jean-Marc Andreoli (2019) https://arxiv.org/abs/1905.01289
Kraj razgovora
Novi razgovor -
-
-
Is Karpathy's multi-head architecture already implementing something similar in a hybrid CNN/Self attention model?
-
Can you share a link to that paper? Thank you
Kraj razgovora
Novi razgovor -
Čini se da učitavanje traje već neko vrijeme.
Twitter je možda preopterećen ili ima kratkotrajnih poteškoća u radu. Pokušajte ponovno ili potražite dodatne informacije u odjeljku Status Twittera.
香港人
⠀
Paper:
Interactive website :
Code:
Blog: