François CholletVerified account

The phenomena I'm explaining are empirical. I like to think about theoretical explanations. The statement "and therefore..." is a non-sequitur, unless I'm missing something.

I think you must be missing something, or I'm not explaining clearly. If you have an un-regularized, very over-parameterized model, trained over a number of epochs, to a global minimum, it won't generalize. That's why we use dropout, data augmentation, weight decay, etc. Right?

No. That's what the community thought. But now it's recognized (though apparently the news hasn't spread) that SGD, on its own, regularizes. Even if you use dropout, batch norm, etc., you may be finding global minima that generalize. Depends which global minima you find!

I'm well aware of that theory. Obviously using mini batches adds randomness that does some regularizing. But not enough in practice for nearly all real world uses. You can use tiny batches to make stronger effect, but that has its own problems.

In practice our best models are nearly always extremely over parameterized, and need a *lot* of regularization. More than just mini batch variation can provide.

I agree that we need regularization in over-parametrized models. But mathematically, this has little to do with the achieved training risk _on its own_. Generalization error can depend on training risk, but also, learned classifier, algorithm, data, data distribution, etc.

Well then we're in agreement

Well then!