Mailchimp's Mandrill App Suffers Service Outage, Company Says. "Transaction ID Wraparound issue", and the database "in read-only mode until offline maintenance (known as vacuuming) can occur" mediapost.com/publications/a
Conversation
I've seen things like this happen before. Usually something like a buggy hand-rolled upsert procedure (with improper handling of duplicate violation) is involved. That can burn through transaction IDs very quickly, without a true spike in writes.
1
1
5
That speculation could be wrong, of course -- it's just something I've observed during similar production outages. Even if I'm right, that's not a great failure mode.
1
1
There are certainly better ones. While I do think that monitoring might have made all the difference here, as an engineer my first thought is "why doesn't buggy code that burns through XIDs senselessly fail in a self-limiting fashion?". Don't know where to go from there, though.