It's interesting to observe how the spread of ideas actually works in practice. I've seen DeOldify cited many places, yet the key insights that to me seem like low hanging fruit don't actually get traction. I suppose not having a paper might be part of it but it's still odd.
-
I am always amused to see papers in 2020 that have very basic training loops that take forever to converge. What is your exp with self attention?
2/ Really when you look at what self-attention is doing more broadly, it's this: Turning a whole much of convolutions to capture a large receptive field into a much smaller set of computations. Self-attention is to convolutions as convolutions are to full connected.
