AI and Compute: Our analysis showing that the amount of compute used in the largest AI training runs has had a doubling period of 3.5 months since 2012 (net increase of 300,000x):https://blog.openai.com/ai-and-compute/
-
-
These are deeply worrying trends: the fact that we need such ridiculous increases in compute for small improvements in performance is a clear indication that we have pushed deep learning as far as it can feasibly go. The diminishing returns of more data and compute are clear.
-
Also, SOTA algorithms in DL are now so incredibly inefficient, requiring such massive distributed computing infrastructures that only companies like Google are able to run them. This is not progress.
- Još 2 druga odgovora
Novi razgovor -
-
-
Extrapolating out these results we find that by 2022 the entire world's compute resources will be used on a single OpenAI project. We may even be able to get a robot hand to solve a rubik's cube without dropping 20% of the time. Isn't the future amazing!
Hvala. Twitter će to iskoristiti za poboljšanje vaše vremenske crte. PoništiPoništi
-
-
-
Please do an analysis of efficiency and knowledge within the community. While compute is definitely a great measure of progress, I believe utilization of the compute could be more indicative of the actual pace of progress.
-
Yes, we think analyzing trends like this on multiple dimensions is really useful, and we hope to produce more analyses that look at different aspects of this
Kraj razgovora
Novi razgovor -
-
-
Amazing. Are there any points above 1 PF/s*day that don't use RL? Last time I looked at these data, all the largest workloads were either RL, or used RL-based training algos -- IIRC GNMT used REINFORCE+SGD for training, but Arxiv strangely seems to have lost the paper.
-
Not plotted here as the new stuff for this analysis is pre-2012, but GPT-2 used something on the order of ~ 10 petaflop days. (My intuition - though note I haven't done calcs here - is a load of LM stuff from this year across AI research is probably quite compute-intensive)
- Još 1 odgovor
Novi razgovor -
-
-
Where would Deep Blue and Watson fit on this chart? They are - surprisingly - missing.
- Još 9 drugih odgovora
Novi razgovor -
-
-
And suddenly carbon footprint matters.
Hvala. Twitter će to iskoristiti za poboljšanje vaše vremenske crte. PoništiPoništi
-
Čini se da učitavanje traje već neko vrijeme.
Twitter je možda preopterećen ili ima kratkotrajnih poteškoća u radu. Pokušajte ponovno ili potražite dodatne informacije u odjeljku Status Twittera.