Conversation

SBF

@SBF_FTX

1) A week ago, we had Cloudflare errors on FTX. I promised a post-mortem. Here it is.

6:08 PM · Jun 23, 2021Twitter Web App

Replying to

2) The context: for about an hour on June 16th, many users got DNS errors when trying to access FTX. That basically means that Cloudflare, our anti-DDOS service, was rejecting a lot of users. You can find the Cloudflare incident here:

cloudflarestatus.com

Elevated number of 502 errors in KIX Osaka

SBF

@SBF_FTX

Jun 23, 2021

3) What happened, roughly: (a) we had a particularly type of logging turned on (b) do to a bug in a maintenance process, Cloudflare accidentally routed logs to one of their shut-down servers (c) this triggered their DDOS systems

SBF

@SBF_FTX

Jun 23, 2021

4) How we reacted: (a) we quickly found out that Cloudflare was blocking users (b) after about 10 minutes, we began switching systems over to a different DNS provider (c) at the same time, we began talking with Cloudflare's team to diagnose the issue

SBF

@SBF_FTX

Jun 23, 2021

5) After about an hour, we fully switched FTX's DNS provider, and service was no longer interrupted. After a few hours, Cloudflare identified and patched the issue; a few hours later, we switched back to Cloudflare.

SBF

@SBF_FTX

Jun 23, 2021

6) I'd like to thank the

@Cloudflare

team for being super responsive and helpful, and patching this same-day. Going forward, we have more practice switching DNS providers and should be able to do this more quickly if there are future issues.