(people were starting to look at each other uneasily and wonder when the Big One™ was gonna hit. happy birthday, @maplebed!)
after a year of prod svc, we finally had our first global outage! here's to many more terrifically fun problems 
https://honeycomb.io/blog/2017/08/our-first-outage/ …
-
-
Thanks. Twitter will use this to make your timeline better. UndoUndo
-
-
-
Nice postmortem. :)
-
thanks!!
- Show replies
New conversation -
-
-
great job, y'all! Ops hug! :-) And thanks for the insightful posting
Thanks. Twitter will use this to make your timeline better. UndoUndo
-
-
-
- thanks for the writeup!Thanks. Twitter will use this to make your timeline better. UndoUndo
-
-
-
Fun! Amusingly schema a have been an occasional hazard for us as well. :)
-
oh boyeee flexible schemas are a world of edge cases. we learned a *lot* about cotenancy problems from parse, lol
- Show replies
New conversation -
-
-
What is the "thundering herd problem", I've never heard that term.
-
https://en.wikipedia.org/wiki/Thundering_herd_problem … when e.g. all the mysql queries are waiting for one to complete, then they all finish at once and bomb the cache
- Show replies
New conversation -
-
-
Less than 2 hours for that kind of mess? Very nice.
Thanks. Twitter will use this to make your timeline better. UndoUndo
-
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.