Greg Yang

4A/ Most commonly, a GP is described as a “Gaussian distribution over a function space.” If the function space has output space R, then a random function f is a GP if {f(x1), …, f(xk)} is jointly Gaussian for any set of inputs {x1, …, xk}.pic.twitter.com/2dZixGQthF

5A/ In this context, “wide, random NN is GP” means that, e.g. if we feed in two different photos into the same random MLP, then we expect the output to be distributed as a Gaussian (with randomness from re-initialization of MLP parameters).pic.twitter.com/wADNydhuCA

6A/ On the other hand, what does it mean to say an RNN is a GP? More generally, we can extend the definition of a GP to function with *variable-dimensional outputs*: such a function outputs f(x) in R^(l(x)), given some length function l.pic.twitter.com/l9Cgbmuit7

7A/ A (word-level) RNN is a function with variable-dimensional output, where an input is a sentence, and an output is a sequence of real numbers, one per input word. The figure here gives an example.pic.twitter.com/dTTuRpy064

8A/ To say that this RNN is a GP is would imply that all outputs of the RNN, over any finite collection of input sentences, are jointly Gaussian, as indicated above. We empirically verify this claim for the last outputs of the RNN after running over each sentence.pic.twitter.com/Wyt0dD8Rdq

9A/ Here’s the kernel describing the joint Gaussian distribution of all the RNN outputs. Note how the same words (e.g. “fox”) in the two sentences induce different output from the network depending on context.pic.twitter.com/Mq2T49owBj

10B/ Now let’s talk about the claim “any architecture.” Our GP limit works for any architecture expressible by a series of matrix multiplications and coordinatewise nonlinearities, with possible reuse of computation results. Such a series is called a “tensor program.”pic.twitter.com/MwDKHT6pbW

11B/ While at face value, this may seem limited, one can in fact express almost all modern architectures in this framework: resnet, transformer, LSTM, etc. Here are some simple examples of tensor programs expressing NN computation.pic.twitter.com/ObYofpPgIw

13/ These are examples of kernels corresponding to the infinite-width limit of other architectures: GRU, transformer, batchnorm.pic.twitter.com/QAQCfWzUOz

14/ Finally, we verify our theory is more and more accurate as the width of the network increases. Here we measure the relative frobenius norm of the empirical kernel to the infinite-width theoretical kernel, and we see that the deviation tends to 0 like 1/sqrt(width)pic.twitter.com/NbOIlejwCq