I am -loving- this combined Rectifed Adam (RAdam) + LookAhead optimizer. It's freaky stable and powerful, especially for my purposes. Only weird thing to keep in mind- I've had to multiply my learning rates by over 100x to maximally take advantage.https://medium.com/@lessw/new-deep-learning-optimizer-ranger-synergistic-combination-of-radam-lookahead-for-the-best-of-2dc83f79a48d …
-
Show this thread
-
Replying to @citnaj @pksmoraes
how about LA Adam on convnets? Image classification community never truly switch to Adam because SGD almost always give 1% more accuracy (important so that you can write papers.)
2 replies 0 retweets 3 likes -
Replying to @tsauri_eecs @pksmoraes
They’re claiming > sgd performance in the paper.
1 reply 0 retweets 0 likes -
Replying to @citnaj @pksmoraes
Then they better thoroughly prove so for various models. If it really has both faster convergence AND accuracy.
1 reply 0 retweets 0 likes
Replying to @tsauri_eecs @pksmoraes
I'm sure they'd appreciate your help. It's an effort going on over there at fastai's forums at that link.
10:08 PM - 25 Aug 2019
0 replies
0 retweets
1 like
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.