grapheneos.org/articles/serve is still working nicely for making the most of our server bandwidth with our greatly expanded infrastructure.
CAKE + net.ipv4.tcp_notsent_lowat gives amazing fair load balancing and latency.
BBRv1 hurts too much and doesn't really seem to provide much...
Conversation
Tuning net.ipv4.tcp_notsent_lowat isn't hard since it doesn't need to be as high under actual load. It's easy to tell it's working from having a lot more context switches since it avoids filling enormous buffers. nginx has no concept of fair scheduling at all so it helps a ton.
1
2
Can set it higher if the server doesn't have CPU cycles to spare but that's pretty much a non-issue for a reverse proxy or static file server.
Might as well be using CAKE and net.ipv4.tcp_notsent_lowat if you're nowhere close to maxing out the CPUs and that's normally the case.
1
3
Fair scheduling means you get great latency even with the bandwidth completely maxed out. It actually gets split quite evenly between not just connections but groups of connections from the same hosts. Works best if you accept more context switches via net.ipv4.tcp_notsent_lowat.
1
2
Without net.ipv4.tcp_notsent_lowat, nginx will be filling enormous kernel buffers until it blocks. It's nice being able to coerce it into not having so much tunnel vision.
Setting it low does require the application is fast enough to keep buffers filled despite switching often.
1
1
Replying to
Out of interest, how low do you have to set it in order to get more fair workloads, and how many active connections is that generally over?
1
Replying to
CAKE with a proper bandwidth limit turning the server or router where it's deployed into the bottleneck is enough to provide very fair scheduling across hosts / connections at the kernel level. Issue is that the send buffers are still enormous so nginx, etc. get tunnel vision.
1
It helps a lot even without net.ipv4.tcp_notsent_lowat and it gradually gets better and better from coercing the applications into switching between connections as you lower it. It's easy to see that lowering it starts causing more context switches + better latency/fairness.
2
You really need net.ipv4.tcp_notsent_lowat on the server itself so you have very low latency feedback to the applications about when the buffers are adequately filled. It just shouldn't be set too low for them to max out the bandwidth which isn't really that hard to tune.
For nginx, it's so quick at waking up and pouring data into the kernel buffers with low CPU usage that setting it to 128k works fine for high throughput, at least near 1-2Gbit. It's also only going to bottleneck very fast connections when the server is under low load regardless.
2
i.e. if you set it to 16k then there's a decent chance that an application may not be able to keep up with a 1Gbit connection consistently. Have a feeling that nginx would normally do fine, but you probably wouldn't really want it below 128k for something serving large files.