OK guys AWS says "a few hours" until they've fixed it.
Meanwhile we're firefighting all of the issues that pop up.
Expect things to be off-and-on.
Conversation
I'm one of your biggest advocates but this needs some additional planning and procedures such as freezing to insert-only when coming back up.
I've got losses because I can't get orders in right now, or they are going in at the wrong price - but some ppls are getting through.
1
3
Replying to
yeah hear you on that -- I guess we could do like 1 minute of post-only whenever it comes back up?
Note that this gets pretty tricky when you start thinking about stops, TPs, liquidations, etc.
1
4
Totally, but its better than skewed book access.
Also points to the need for some hot switch ability across multiple infra providers.
I know a few SREs who have worked on crypto exchanges & large finance product reliability. If your team needs support/insights let me know.
1
2
Never understood why exchanges don’t have mirrors, is there a reason or is it just a bit more complicated than I think?
1
There is complexity, especially depending on what part is hit, but I had it on the exchange I was a director of in 2014 so it is doable and it’s necessary given the volume of money.
The big mistake some make is not having it across diverse providers.
1
4
yeah def doable, just takes time; these things tend to be a game of whack-a-mole