Some more detailed reflections on "The Bitter Lesson": cognitivemedium.com/bitter-lesson
Conversation
Replying to
I've been thinking about this all day, and one thing that I can't get past is that Moore's law (and more recently the advances in deep learning-specific hardware) didn't happen through some natural process: we made them happen!
1
I don't know very much about hardware, but every time I dip my toes in (e.g., take a class, read some papers) I'm struck by how much human effort goes into making our hardware ever faster. This seems to involve lots of people thinking very hard about computation.
I wonder if the bitter lesson should involve the place where we wanted to apply the brain-sweat? Instead of grand ideas about the nature of human knowledge, make linear algebra really really fast?
2