michael_nielsen

Good luck trying to do deep learning without convolutions, LSTMs, ReLUs, batch normalisation, etc. Good luck trying to solve Go without the prior knowledge that the problem is stationary, zero sum, and fully observable.

So the history of AI is not the story of the failure to incorporate human knowledge. On the contrary, it is the story of the success of doing so, achieved through an entirely conventional research strategy: try many things and discard the 99% that fail.

The 1% that remain are as crucial to the success of modern AI as the massive computational resources on which it also relies.

Sutton says that the intrinsic complexity of the world means we shouldn’t build prior knowledge into our systems. But I conclude the exact opposite: that complexity leads to crippling intractability for the search and learning approaches on which Sutton proposes to rely.

Only with the right prior knowledge, the right inductive biases, can we ever get a handle on that complexity.

He says “Modern deep-learning neural networks use only the notions of convolution and certain kinds of invariances, and perform much better”. The use of the word ‘only’ highlights the arbitrariness of the claim.

Deep learning wouldn’t succeed without those convolutions and invariances but these are deemed minimal and general enough to be acceptable.

In this way, “The Bitter Lesson” avoids the main question, which is not WHETHER to incorporate human knowledge (because the answer is trivially yes) but WHAT that knowledge should be and WHEN and HOW to use it.