Paolo ArdoinoVerified account

1/ @bitfinex infrastructure thread Till Jan 2018 Bitfinex was using cloud hosting providers (AWS) as main site for the platform. We soon figure out that was almost impossible to scale a CEX on cloud. Here is the train of thoughts that we had in the mighty infrastructure team.

2/ Problems (1): - order/execution full roundtrip should be ~1ms or lower, should be improved over time (0.1ms target) - mange 10k+ to 100k+ orders/sec - recording to historical data stores all events, available to users via API and UI

3/ Problems (2): - handling billions of events per day and recording them it's one of the main bottlenecks of (crypto) exchanges -> DBMS have to record 100k events per second sustained? - on cloud you're not sure how CPU gets allocated (trading engine is a CPU bound process)

4/ Problems (3): - disk throughput can be throttled (GP2, IO1) - if you use SSDs on cloud, which model do you have, are you the only writer?, how to get a proper RAID setup?, ... - how to minimize the number of switches between your instances? - unpredictable network behaviors

5/ Problems (4): - reliable UDP multicast: AWS seems not to support UDP multicast, while GC does not. Anyway what's the expected packet loss? - security: spectre attacks, data hosted on 3rd party, ... - #crypto is growing. When next bull run? How to ensure 10x/100x scalability

6/ Path to metal Our infra team has outstanding experience in secure, god wrath resistant, deployments, so we discussed a lot about pros and cons. Here are the 2 cons summarized: - hardware decay - physical hosts maintenance (trips to datacenter, hardware expertise, ...)

7/ Benefit (1): - the entire infra behavior is predictable - RAID enabled historical sharded data-warehouses kick asses - KYD, KYM and KYC : Know Your Disk, Know your Mem and Know Your CPU - dedicated switches and firewalls, back to back connectivity for core servers