🔥 🔥 🔥 💻 💻 💻 💻 🔥 🔥 🔥

This is impressive work. I have so many (immediate) questions, but less on the side of RL. I'd really like to understand how the team went about operationalising this system.

How did they measure their progress? Did they write any tests (unit, integration, other) to check their working? What did they use to monitor it? Could they jump into strace to debug the running system and "change" it's path? Did they use distributed tracing or just images?

How easy was it to dump the state of the system to do point-in-time recovery? How are they making it cost effective? How much of the system is actually utilised vs. idle?

What research are they doing to make proceeding training runs and experiments cheaper to run? Are TPUs worth it or are there better hardware paths that could be taken? How much of Google's funding is dependent on @deepmind falling in line with certain corporate goals?

Been a lot of discussion around training cost recently, especially with Timnit and her paper. This always surprised me, as training requires less carbon than even individual consumer decisions like buying a car In your view, is the issue more reproducibility?

The carbon point is interesting. If you control your hardware setup, you can track the electricity draw of your cage/row of servers and then offset that by planting more trees. This is something Deepmind, GCP, AWS, and the other big players can do. It's not viable for others.

AFAIK Google is already carbon neutral (for more than a decade now). I think they are now trying to move into carbon negative so this concern doesn't quite apply (doubt it's this simple so take that with a grain of salt).

As you're both here, I'm looking for more papers to read. Got any recommendations or things you're interested in but haven't read yet? The papers being behind a paywall isn't an issue.