You can add location information to your Tweets, such as your city or precise location, from the web and via third-party applications. You always have the option to delete your Tweet location history. Learn more
The @fastdotai community has been hard at work trying to make sense of the new state of the art in optimizers. Towards this end @mgrankin put together this repo with results + implementations! I'll definitely be trying this. https://github.com/mgrankin/over9000 …pic.twitter.com/wfKd8MWp57
Note that there's still questions around what learning rate schedule is appropriate when warmup is apparently obsolete. So far I've done flat lr then decay, but that was just an educated first guess. See discussion:https://forums.fast.ai/t/imagenette-woof-leaderboards-guidelines-for-proving-new-high-scores/52714/19 …
Where is SGD momentum baseline.
What scheduler would you recommend for Over9000? I've seen CLR's effectiveness with SGD, but not sure with the combination of three.
For now the consensus seems to be flat lr then cosine annealing.
Also Lookahead : 0.61 Lookahead+RAdam: 0.595 LARS+RAdam : 0.574 Lookahead+Radam+LARS: 0.645 (??) So the gains from these optimizers are in some sense orthogonal and presumably from the (Lookahead+LARS) combo, both of which alter the entire training dynamics (unlike RAdam).
What does RAdam + LARS + Lookahead mean? You ran these optimisers one after the other or are they some tricks you used all at once?
The three are combined together, Over9000.https://github.com/mgrankin/over9000 …
Since results are reporting averages of many runs, would be great to include CIs around those estimates.
You can find calculated STD in the notebooks.
Thank you
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.
