Paolo ArdoinoVerified account

Among the many benefits of running @bitfinex outside of AWS (predictability of the system, performance, ...), we were sick tired of AWS rebooting instances without much of a notice. In many occasions AWS sent a communication after the fact.

If you can’t build your systems in a way that is so resilient to this that you don’t even notice... then you need a new engineering team.

Nah. @bitfinex has the best engineering team. In fact is almost the only exchange always up during windy markets. Our infra has replica, sharding and 100% resiliency. I just don't like random reboots. Dealing with high-frequency trading is not as dealing with an e-commerce

If you don’t like random reboots your system is brittle. Even if they didn’t happen, you should arguably be triggering them randomly anyway to make sure your system is robust. This is what proper engineering teams do https://netflix.github.io/chaosmonkey/ 

Do you understand that "not liking random reboots" is different from "not being prepared" right?

Ok so what is the reason for your dislike?

If you have random reboots, especially on the matching engine, a single truth source for exchanges, you can of course fail over, but there is a lot of follow-up checking to do to ensure no message got lost etc. When you deal with people's money, predictability is important.

In other words, you designed a system that is too brittle to run on AWS reliably.