1/ @bitfinex infrastructure thread
Till Jan 2018 Bitfinex was using cloud hosting providers (AWS) as main site for the platform.
We soon figure out that was almost impossible to scale a CEX on cloud.
Here is the train of thoughts that we had in the mighty infrastructure team.
-
-
3/ Problems (2): - handling billions of events per day and recording them it's one of the main bottlenecks of (crypto) exchanges -> DBMS have to record 100k events per second sustained? - on cloud you're not sure how CPU gets allocated (trading engine is a CPU bound process)
Show this thread -
4/ Problems (3): - disk throughput can be throttled (GP2, IO1) - if you use SSDs on cloud, which model do you have, are you the only writer?, how to get a proper RAID setup?, ... - how to minimize the number of switches between your instances? - unpredictable network behaviors
Show this thread -
5/ Problems (4): - reliable UDP multicast: AWS seems not to support UDP multicast, while GC does not. Anyway what's the expected packet loss? - security: spectre attacks, data hosted on 3rd party, ... -
#crypto is growing. When next bull run? How to ensure 10x/100x scalabilityShow this thread -
6/ Path to metal


Our infra team has outstanding experience in secure, god wrath resistant, deployments, so we discussed a lot about pros and cons.
Here are the 2 cons summarized:
- hardware decay
- physical hosts maintenance (trips to datacenter, hardware expertise, ...)Show this thread -
7/ Benefit (1): - the entire infra behavior is predictable - RAID enabled historical sharded data-warehouses kick asses - KYD, KYM and KYC
: Know Your Disk, Know your Mem and Know Your CPU
- dedicated switches and firewalls, back to back connectivity for core serversShow this thread -
8/ Benefit (2): - reliable UDP multicast - bare-metal Linux installs - you're the only one using your CPU (see spectre-like attacks) - colocation services offering
Show this thread -
-
New conversation -
-
-
Thanks. Twitter will use this to make your timeline better. UndoUndo
-
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.