Możesz dodawać lokalizację do Twoich Tweetów, jak miasto czy konkretne miejsce, z sieci lub innych aplikacji. W każdej chwili możesz usunąć historię lokalizacji swoich Tweetów. Dowiedz się więcej
Q. Can we solve learning dynamics of modern deep learning models trained on large datasets?
A. Yes, by combining symmetry and modified equation analysis!
co-led with @KuninDaniel (now on twitter)
& @jvrsgsty @SuryaGanguli @dyamins
Neural Mechanics https://arxiv.org/abs/2012.04728
1/8pic.twitter.com/5jRw8AbUsm
To study complex learning dynamics of neural networks, existing works made major assumptions (i.e. single hidden layer, linear networks, infinite width) Instead, we uncover combinations of parameters with simplified dynamics that we solved exactly without a single assumption 2/8pic.twitter.com/8579HCprP3
Gradients and Hessians, at all points in training, obey geometric constraints due to symmetry A network has a symmetry if the loss doesn’t change under a transformation of the parameters (i.e. translation, scale, rescale for parameters preceding softmax, batchnorm, ReLU) 3/8pic.twitter.com/W71whnaBL8
Every symmetry of a network has a corresponding conserved quantity through training under gradient flow (Noether's theorem for neural networks!) For translation, scale, and rescale symmetry the flow is constrained to a hyperplane, sphere, and hyperbola respectively 4/8pic.twitter.com/ROn64m52xu
Gradient flow is too simple for realistic SGD training. We construct a more realistic model considering weight decay, momentum, mini-batches, and a finite learning rate We use modified equation analysis to model the effect of discretization (as also done in recent works) 5/8pic.twitter.com/45qfmL8lfo
The realistic model for SGD breaks the conservation laws of gradient flow, resulting in simple first and second order ODEs We can solve these ODEs exactly leading to theoretical solutions we empirically verify on VGG-16 training on Tiny ImageNet 6/8pic.twitter.com/hpQiP4vNsX
These solutions confirm existing phenomenon, such as the spherical motion of parameters before batch normalization, while highlighting new phenomenon, such as the harmonic motion of parameters before the softmax function 7/8pic.twitter.com/WdZ7e7OXq6
Overall, our work provides a first step towards understanding the mechanics of learning in neural networks without unrealistic simplifying assumptions Check out the paper for more details: https://arxiv.org/abs/2012.04728 8/8
I'm familiar with backward error analysis. Can you explain how that interacts with the symmetry considerations?
The symmetry yields additional constraints on gradients and Hessians that allow you, among other things, to project out sgd noise and obtain simple exact dynamic learning laws for parameter combinations that respect the symmetry
Congrats!
Twitter jest przeciążony lub wystąpił chwilowy problem. Spróbuj ponownie lub sprawdź status Twittera, aby uzyskać więcej informacji.
