🔥 🔥 🔥 💻 💻 💻 💻 🔥 🔥 🔥

Take a sec to think about how many new grads/interns are on the FB eng team at any given time, and marvel anew at this act of trust

... trust in the *tools*, I mean; not the interns. lol. feature flags, waves and waves of canaries and rolling deploys and A/B testing,

FB goes down for the same reasons the rest of us do: they deploy bad code, especially configs (configs are code too!)

They solve this by running an elaborate staging environment, using two globally distributed data centers and a snapshot of real data...

Just fucking kidding, of course. They use all their fancy-ass tools to roll lots fast and stage small user slices and TEST IN PRODUCTION.

I do miss sandcastle, the fleet that would run a bajillion tests and auto-accept your diff hours later if they all ran cleanly