- make it possible - make it work - make it efficient - make it dependable and invisible - move on to the next layer and never think about it again
Show this thread
-
We're still exploring the space of problems we can solve, but I think that was mostly 2014-2016 (while 2011-2013 was "wow, it works"). DL is now a well-established solution for a certain scope of problems, but still a very inefficient one (data-inefficient, power-inefficient...)
Constantly retraining from scratch very similar models, taking days at a time (sometimes weeks or months) seems particularly wasteful. At a lower-level, most models seem to use way more ops/params than they need, and backprop + SGD itself is fundamentally inefficient
-
-
Nods. And that I think is consistent with my view that we haven’t necessarily figured out the optimal architecture...we grok their general shape, but they are not necessarily efficient,
-
Efficiency is the enemy of the good. It’s because they’re inefficient that they’re resilient to damage and good at combining approximations to a better result. This is exactly why the brain is good - it makes our flaws look like genius, but we’re just stumbling along
End of conversation
New conversation -
-
-
True. I believe Hinton would concur. Took his course on edx, he talks about backprop being inefficient etc on the discussion forums. Unfortunately, people aren't taking him seriously regarding this.
Thanks. Twitter will use this to make your timeline better. UndoUndo
-
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.