grapheneos.org/articles/serve is still working nicely for making the most of our server bandwidth with our greatly expanded infrastructure.
CAKE + net.ipv4.tcp_notsent_lowat gives amazing fair load balancing and latency.
BBRv1 hurts too much and doesn't really seem to provide much...
Conversation
Tuning net.ipv4.tcp_notsent_lowat isn't hard since it doesn't need to be as high under actual load. It's easy to tell it's working from having a lot more context switches since it avoids filling enormous buffers. nginx has no concept of fair scheduling at all so it helps a ton.
1
2
Can set it higher if the server doesn't have CPU cycles to spare but that's pretty much a non-issue for a reverse proxy or static file server.
Might as well be using CAKE and net.ipv4.tcp_notsent_lowat if you're nowhere close to maxing out the CPUs and that's normally the case.
1
3
Fair scheduling means you get great latency even with the bandwidth completely maxed out. It actually gets split quite evenly between not just connections but groups of connections from the same hosts. Works best if you accept more context switches via net.ipv4.tcp_notsent_lowat.
1
2
Without net.ipv4.tcp_notsent_lowat, nginx will be filling enormous kernel buffers until it blocks. It's nice being able to coerce it into not having so much tunnel vision.
Setting it low does require the application is fast enough to keep buffers filled despite switching often.
1
1
Replying to
Out of interest, how low do you have to set it in order to get more fair workloads, and how many active connections is that generally over?
1
Replying to
CAKE with a proper bandwidth limit turning the server or router where it's deployed into the bottleneck is enough to provide very fair scheduling across hosts / connections at the kernel level. Issue is that the send buffers are still enormous so nginx, etc. get tunnel vision.
1
It helps a lot even without net.ipv4.tcp_notsent_lowat and it gradually gets better and better from coercing the applications into switching between connections as you lower it. It's easy to see that lowering it starts causing more context switches + better latency/fairness.
2
Replying to
right, so a big part of the fairness issue is actually in the rotation through buffers already in-kernel, by mediating their size, you're causing them to be consumed more fairly. I wonder if this is related to the connection hash bucketing, have you looked at tuning ethtool?
1
Replying to
CAKE provides fair balancing across sources, destinations and connections themselves. It's really good at doing it. You need CAKE at the actual bottleneck where buffers are getting filled up so ideally you can properly set a bandwidth cap for it without the real cap fluctuating.
If you have a stable 1Gbit connection and set cap to 1Gbit or a little bit below, it does truly amazing traffic shaping / fair balancing. If there ends up being a bottleneck elsewhere, then that router ends up being where buffers fill and you lose most fairness for that portion.
1
Replying to
right, i run cake here, but for example the intel nic driver i'm using hosts two channels, but they don't actually balance properly, and the best thing to do to make cake work properly is to disable one of the channels, either by entirely disabling it, or by no-match bucket