The amount of compute thrown at popular AI training runs has increased 300,000x since 2012. 1/5 the doubling time of Moore's Law.https://blog.openai.com/ai-and-compute/
-
-
Is there a clear-cut system for distinguishing between different orders-of-evaluation in training runs?
-
what i mean is that in eg theory of the firm you have fixed (extrinsic) costs, fixed operating costs, and per unit (marginal) costs
-
likewise in using of computing power you have computing power X used to eg play a game of go... then you have compute Y used to generate heuristics used to play go at cost X... and Z to gen heuristic for heuristic-generation...
-
so long as the challenge is just “can an AI do this task, ever?” it doesn’t matter that much but to make extrapolations about AI use from Moore’s Law you need the distinction
-
Yes, model archetecture still constrains the bounds. But within that picture of plot of X_training run size * Y_generalization error..
-
..it's some kind of s-curve where the gains from X dont seem to be asymptoting. (I dont know enough about the field to say why)
-
Tweet unavailable
-
tantum needs to accept my follo i cant see this conversation
- 1 more reply
New conversation -
-
-
It means many of the gains from this (statistical / ML) approach are driven by increases in computational resources devoted to the task rather than increases in the "smartness" of the algorithms they are running
Thanks. Twitter will use this to make your timeline better. UndoUndo
-
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.