Conversation

Elevated number of 502 errors in KIX Osaka

2) The context: for about an hour on June 16th, many users got DNS errors when trying to access FTX. That basically means that Cloudflare, our anti-DDOS service, was rejecting a lot of users. You can find the Cloudflare incident here:

cloudflarestatus.com

3) What happened, roughly: (a) we had a particularly type of logging turned on (b) do to a bug in a maintenance process, Cloudflare accidentally routed logs to one of their shut-down servers (c) this triggered their DDOS systems

6:08 PM · Jun 23, 2021Twitter Web App

4) How we reacted: (a) we quickly found out that Cloudflare was blocking users (b) after about 10 minutes, we began switching systems over to a different DNS provider (c) at the same time, we began talking with Cloudflare's team to diagnose the issue

Replying to

5) After about an hour, we fully switched FTX's DNS provider, and service was no longer interrupted. After a few hours, Cloudflare identified and patched the issue; a few hours later, we switched back to Cloudflare.

6) I'd like to thank the

@Cloudflare

team for being super responsive and helpful, and patching this same-day. Going forward, we have more practice switching DNS providers and should be able to do this more quickly if there are future issues.