How I learned of the importance of idempotency for work queues: a thread.https://twitter.com/ElleArmageddon/status/1255870742727585792 …
-
-
What I thought had happened: OK, X,000 less important analytics events and a few dozen phone calls scheduled for this morning delivered within a half hour of scheduled time. That was not what had happened.
Show this thread -
What had actually happened was, every five minutes for 12 hours, an entry was added to the queue for each person who needed a phone call. When I restarted the queue, they all fired within 2 minutes.
Show this thread -
Now let’s talk about the scalability or cloud systems versus the scalability of a plain old telephone. The telecom system is very capable of delivering 80 phone calls in 2 minutes. Phones don’t tolerate that so well.
Show this thread -
So this manifested for customers’ customers as their phones ringing off the hook. Answer a call. Hang up on it. That’s one. Phone rings immediately. Answer and hang up. That’s two. Most people pulled the cord from the wall before counting to ~70.
Show this thread -
When I realized what had happened it was after midnight in Japan. I was in a new apartment with no reliable phone or Internet. I walked across town, in freezing rain, holding a laptop and wired phone, so that I could deliver apology calls to customers and customers’ customers.
Show this thread -
I broke down crying after the first three, and called my dad, certain that I had just bankrupted the business. We actually lost two customers. One came back after receiving an explanation. Anyhow, that’s why you add to queues using an idempotency key.
Show this thread -
Worth noting, since people often think that only noobs made mistakes: I was 10 years into my career at this point, have a CS degree, had run multiple businesses and written code used in production by Serious People at my day job, etc.
Show this thread
End of conversation
New conversation -
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.