I don't recall if there has been any rigorous study about whether to do conv-act-bn or conv-bn-act (I always do conv-bn-act since it's the most intuitive). Anyone has a good reference?
-
my personal workflow is: try to remember the right answer, use google, re-read this one stackoverflow comment quoting you quoting szegedy, remember it is unconvincing, test both variations
https://stackoverflow.com/questions/39691902/ordering-of-batch-normalization-and-dropout#comment78277697_40295999 …pic.twitter.com/6CFOKzClZf
Yeah, the whole field makes no sense. None of us even knows why BN works in the first place (the original narrative "it reduces covariate shift", while intuitive, is almost certainly wrong)
-
-
ML explanations as to why things work are often a little unconvincing. Too often people just pretend that everything makes perfect intuitive sense.
ML seems more of an empirical science than a science of proofs.
We are getting better intuitions & rules of thumb w/ time.Thanks. Twitter will use this to make your timeline better. UndoUndo
-
-
-
So true! Every time my prof, colleagues and online tutorials explain the BN in that way makes me feel frustrating and I never feel satisfied with an answer so far. Finally find someone can echo me on this point.
Thanks. Twitter will use this to make your timeline better. UndoUndo
-
-
-
There was a paper that came out last year that talks about the graident landscape and the effect of BN on it. Lemme find the link to that paper
- End of conversation
New conversation -
-
-
I feel like this one is pretty good hypothesis for BN in ResNets, but obviously there's still a whole lot to explain besides ResNetshttps://arxiv.org/abs/2002.10444
Thanks. Twitter will use this to make your timeline better. UndoUndo
-
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.