Możesz dodawać lokalizację do Twoich Tweetów, jak miasto czy konkretne miejsce, z sieci lub innych aplikacji. W każdej chwili możesz usunąć historię lokalizacji swoich Tweetów. Dowiedz się więcej
We build a probabilistic gradient dynamics model, and explore inference as a sub-routine *for* stochastic optimization!
Check out our contributed talk at …https://i-cant-believe-its-not-better.github.io (Dec 12)
Paper: https://arxiv.org/abs/2011.04803
w/ @damichoi95 @lukas_balles @DavidDuvenaud @PhilippHennig5pic.twitter.com/Ca66L9emWF
We designed a gradient dynamics model, using quantities such as the _variance_ of minibatch gradients and Hessian-vector products. (Done with vmap in #JAX.)
We then infer the true gradient conditioned on a history of stochastic gradients. i.e. p(∇f_t | g_1, ..., g_t)pic.twitter.com/dqRsj5J2hC
Inference under our model is equivalent to Kalman filtering. Curvature-corrected momentum-like updates naturally appears, and the variances determine how much we should trust our gradient samples. This results in an online variance reduction algorithm.pic.twitter.com/HtKAlewHhg
What else can we do with the variance estimates at hand? We can build a 1D Gaussian process for estimating the objective function, in any descent direction. Adaptive step sizes can be chosen to trade-off between minimization and uncertainty!pic.twitter.com/yOmGFm0Ylt
Our approach passes all the unit tests. But ultimately, it comes up short in deep learning optimization. The problem? Because it works! It's diving straight into high-variance high-curvature regions, but doesn't explore enough to find better minima.pic.twitter.com/cDQXJcddv4
Exciting work! I’ve got very relevant prior work at the main conf, showing that Bayesian filtering recovers Adam(W) and performs better than other adaptive NN optimisers! (https://arxiv.org/abs/1807.07540 )
Thanks for the heads up! Nice to see progress on similarities between iterative parameter inference and existing optimization algorithms. Our work centers around gradient inference and automating hyperparameter schedules, but we'll add a discussion around parameter inference.
Is the video of the talk published somewhere for people not registered at NeurIPS 2020?
Twitter jest przeciążony lub wystąpił chwilowy problem. Spróbuj ponownie lub sprawdź status Twittera, aby uzyskać więcej informacji.
