François CholletVerified account

Can differentiable programming learn arbitrarily complex behaviors, up to superhuman performance, given a dense sampling of the behavior manifold (e.g. an infinite training data generator, such as self-play)? Yes. We knew that. Arguably AlphaGo was the concrete proof-of-concept.

Thus, the question of failure or success boils down to "have we drawn enough training data yet?" (or in some cases, "does our architecture have sufficient capacity & appropriate structure wrt the training data available?"). Whether the answer is yes or no, it teaches us nothing.

Note that the question of model choice (the structure of the differentiable architecture used) is rather secondary, because as long as memorization capacity is sufficient, *any* model will do -- if there is training data to match. Even a single hidden Dense layer.