Well there was the time when all the session ids hashed to zero so every time everyone refreshed the page they were logged in as a different user
-
Then there was the biweekly "blame Flock" because its metrics spiked during every overload event, whether internal or external, merely because everything queried it
And of course my manual delivery of Demi Moore and Ashton's tweets during Oprah's live show because intersections between two popular users would time out. @goldman did some very strategic PM-ing by watching the show and saying TWEET in the chat.
-
-
Yeah maybe....many years ago, I told the Oprah story to our boy
@finkd -
When we first parallelized tweet fanout, each time
@ladygaga tweeted the entire site blocked for 20 seconds delivering her tweet only. Then it caught up on everybody else. I was thrilled because it worked as designed but everybody else was upset. -
Tweet unavailable
-
They weren't necessarily harder, but they were definitely stupider
-
A common pattern for each service migration was to hit a data integrity mismatch during rollout and after exhaustive investigation discover that the original service had an edge case that was irrecoverably corrupt
-
Maybe 50% of these scenarios were caused by failing to change the default character encoding
-
Later on there was a Ruby stack corruption issue because the size of the linked libraries (?) exceeded some unchecked threshold and caused constant segfaults; we tried to debug the application for a week until
@bitbckt got lower level tools and figured it out. - 11 more replies
New conversation -
-
-
So clutch. I also spent a year or so saying “it’s ok to flush the queue” in chat. Those were my two jobs.
-
Yeah I think
@robey spent the most time on queue flush duty by far -
I remember needing to update our total user count for a board deck and we had no dashboard to reference so someone had to run a query on a production database in the middle of the night, otherwise we’d bring the site down.
-
Oh yeah, the mythical executive dashboard that "some contractors" were working for Twitter's entire existence. I assume this project is still going on and still unshipped.
-
Wasn’t this birdbrain?
-
No...for a while there was a project management tool we tossed together with DabbleDB though.
-
You wrote such great notes about that and we never published the case study.
- 1 more reply
New conversation -
-
-
Didn’t we also have a fake shadow site setup during that show? Incase it failed...
-
Wasn’t fake, was isolated instance.
-
Yeah, that was it. Sorry to imply it was fake somehow...
-
I’m just pedantic you know
-
you're missing a comma after the third word
End of conversation
New conversation -
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.