As someone who has never built a system that had to account for serious backpressure, I am 100% sure it will bite me in the ass in the near future, no matter how seriously I try to control for it. This definitely feels like one of those concepts you pay a tuition in pain for.
-
-
Replying to @generativist
Yes, it’s easy to get bitten because it’s a behavior of the entire system. Not something you can catch with unit tests or even integration tests. Some common contributors to failure modes here: * queues of unbounded length * timeouts that are set too high
2 replies 0 retweets 5 likes -
Replying to @lhochstein
Thanks for the tips, I know this is your area! TBH I think I may even need to start a little closer to fundamentals with instrumentation. What I have built previously and what I'm trying to build now have such *vastly* different costs in terms of failing to log and watch.
3 replies 0 retweets 2 likes -
Replying to @generativist @lhochstein
Get your instrumentation in there. That plus some observability will help. Then adjust load levels up gradually until you see stuff breaking. (Idk, hard area, but those are my thoughts.)
1 reply 0 retweets 1 like
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.
