It's interesting to observe how the spread of ideas actually works in practice. I've seen DeOldify cited many places, yet the key insights that to me seem like low hanging fruit don't actually get traction. I suppose not having a paper might be part of it but it's still odd.
-
I am always amused to see papers in 2020 that have very basic training loops that take forever to converge. What is your exp with self attention?
1/ Well I can tell you this much: I've had several instances where I've accidentally disabled self attention, and I'm alerted to it because it makes such a visual difference. Namely- large details lose consistency (like stretches of forest or skin) and small details are missed.
-
-
have you tried replacing the convnets by a full self attention model, like the ones on the Axial Attention papers?
-
I haven't yet but it sounds like quite a reasonable idea.
End of conversation
New conversation -
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.