#DeepLearning folks, remember AMSGrad? Has it worked better than plain Adam for you lately?
-
-
Replying to @deliprao
Adam has been my default and typically I don’t try anything else. I thought all the other crazy optimizers (not AMSgrad) are just there for people chasing SOTA... do any actually consistently work bettter?
2 replies 2 retweets 32 likes -
Replying to @dennybritz @deliprao
Good question if Adam works consistently better than SGD (or SGD with momentum).
1 reply 0 retweets 0 likes -
At least I don’t need to tune Adam, but you need to tune SGD + momentum, right?
1 reply 0 retweets 3 likes -
Replying to @dennybritz @deliprao
Well, as FastAI recently showed, you do. Also, pure SGD seems to work better than people think. Adam seems to be good for experimentation (very fast convergence), but does it generalize as well as SGD? I have some doubts.
2 replies 0 retweets 9 likes -
Replying to @srchvrs @dennybritz
Well, nothing generalizes well as a *carefully* tuned SGD.
2 replies 0 retweets 5 likes -
That's not generally true any more
2 replies 0 retweets 0 likes -
link? SGD has capacity to discover optimal parameters while other variants of SGD we trade off some of that capacity for convenience.
2 replies 0 retweets 6 likes -
See the experiments i linked to earlier . Since it isn't a PDF no one seems to bother reading it or citing it, but it's the most complete review I know of http://www.fast.ai/2018/07/02/adam-weight-decay/ …
6 replies 13 retweets 112 likes -
Replying to @jeremyphoward @deliprao and
People are happy to cite non-pdf. But cueing them helps, egpic.twitter.com/W6yLvJOFvO
1 reply 0 retweets 15 likes
And: (Note: DOIs seem to be helpful, but I didn't get one for my book. This means it's not indexed in some places, but mostly doesn't seem to have hurt. http://Distill.pub gets them for everything, though.)pic.twitter.com/hmPP0kSQkU
-
-
Replying to @michael_nielsen @jeremyphoward and
Also inclined to believe, as Ian says, that if you expect to get credit in accord with standard norms, it helps to participate yourself in those norms, eg:pic.twitter.com/ScfESB2I1b
3 replies 0 retweets 5 likes -
Replying to @michael_nielsen @jeremyphoward and
It's frustrating how opaque academic norms are to outsiders - it's not like there's a manual of this stuff. And I'm sure I violate some ML community norms, as an outsider to that community. But I'd guess those norms are the obstacle here, not some animosity toward pdf or fastai
1 reply 0 retweets 9 likes - 8 more replies
New conversation -
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.