Conversation

1) A week ago, we had Cloudflare errors on FTX. I promised a post-mortem. Here it is.

9

20

280

2) The context: for about an hour on June 16th, many users got DNS errors when trying to access FTX. That basically means that Cloudflare, our anti-DDOS service, was rejecting a lot of users. You can find the Cloudflare incident here:

cloudflarestatus.com

Elevated number of 502 errors in KIX Osaka

1

40

3) What happened, roughly: (a) we had a particularly type of logging turned on (b) do to a bug in a maintenance process, Cloudflare accidentally routed logs to one of their shut-down servers (c) this triggered their DDOS systems

1

40

4) How we reacted: (a) we quickly found out that Cloudflare was blocking users (b) after about 10 minutes, we began switching systems over to a different DNS provider (c) at the same time, we began talking with Cloudflare's team to diagnose the issue

2

36

5) After about an hour, we fully switched FTX's DNS provider, and service was no longer interrupted. After a few hours, Cloudflare identified and patched the issue; a few hours later, we switched back to Cloudflare.

2

1

38

6) I'd like to thank the

team for being super responsive and helpful, and patching this same-day. Going forward, we have more practice switching DNS providers and should be able to do this more quickly if there are future issues.

2

1

87

7) Finally I'd like to apologize to Cloudflare and to our developers for the extent to which I'm sure I've botched this explanation.

6:08 PM · Jun 23, 2021Twitter Web App

Replying to

and

We love u Sam

1

Replying to

and

Laughing Funny GIF

GIF

read image description

ALT

Replying to

and

Wow, sounds just like what happened to Pancake Swap around March I think it was. I like that you stuck to your word on the debrief 🍻😎✌️☯️

1

Satoshi ₳lien

Replying to

and

You should be using fastly instead

Replying to

and

Hey Sam, what's going on with these Huobi transfers to FTX?

1

Replying to

and

What were you logging 🤔

Replying to

and

Props to you on following up 😎

Replying to

and

so Cloudflare is your DNS + CDN MAIN provider, and now you have second "BACKUP" DNS CDN provider, which I assume AWS Cloudfront or Fastly?