For saddle point problems and more general games, gradient descent can cycle and diverge. Using an extrapolation steps adresses this issue and converges to a saddle point (Nash equilibrium). https://en.wikipedia.org/wiki/Variational_inequality …pic.twitter.com/Kpxbd2Cqbh
We have a new method called competitve gradient descent that uses mixed hessian. Unlike extra gradients these can work with high learning rateshttps://twitter.com/AnimaAnandkumar/status/1205173860284293121?s=20 …
