Dan Roberts

I'm delighted to share publicly "The Principles of Deep Learning Theory," co-written with @ShoYaida, and based on work also with @BorisHanin. It will appear on the @arxiv on Sunday and will be published by @CambridgeUP early next year: https://deeplearningtheory.com/  1/

PDLT develops an effective theory approach to understanding deep neural networks of practical relevance. From a first-principles component-level picture of networks, we explain how to determine an accurate description of the output of trained networks. 2/

A main result is that the predictions of fully-trained networks are described by nearly-Gaussian distributions, with the depth-to-width aspect ratio of the network controlling the deviations from the infinite-width Gaussian description. 3/

With these tools, we can learn in detail about the inductive bias of architectures, hyperparameters, and optimizers. 4/

To obtain these results, we develop the notion of a representation group flow (RG flow) to characterize the propagation of signals through the network. By tuning the networks to criticality, we give a practical solution to the exploding and vanishing gradient problem. 5/

To account for different nonlinear activation functions, we show how RG flow leads to near-universal behavior and lets us categorize them into universality classes. 6/

We explain how these effectively-deep networks learn nontrivial representations from training and more broadly analyze the mechanism of representation learning for nonlinear or weakly-interacting machine learning models. 7/

From a nearly-kernel-methods perspective, we find that the dependence of such models' predictions on the underlying learning algorithm can be expressed in a simple and universal way. 8/