All right, ops types: what’s the biggest production blunder you’ve made?
oh boy! we had an active/standby setup and had to do a failover between them. it went poorly, so we failed back (with a brief period of data loss). But didn't disable a cron job on the standby, so the next morning a cron ran in BOTH places...
-
-
... charging our customers' CUSTOMERS' credit cards twice for every transaction that was queued up. We managed to void all the transactions before settlement, but communicating to indirect customers was not something we were used to doing. Not a good scene.
Thanks. Twitter will use this to make your timeline better. UndoUndo
-
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.