grapheneos.org/articles/serve is still working nicely for making the most of our server bandwidth with our greatly expanded infrastructure.
CAKE + net.ipv4.tcp_notsent_lowat gives amazing fair load balancing and latency.
BBRv1 hurts too much and doesn't really seem to provide much...
Conversation
Tuning net.ipv4.tcp_notsent_lowat isn't hard since it doesn't need to be as high under actual load. It's easy to tell it's working from having a lot more context switches since it avoids filling enormous buffers. nginx has no concept of fair scheduling at all so it helps a ton.
1
2
Can set it higher if the server doesn't have CPU cycles to spare but that's pretty much a non-issue for a reverse proxy or static file server.
Might as well be using CAKE and net.ipv4.tcp_notsent_lowat if you're nowhere close to maxing out the CPUs and that's normally the case.
1
3
Fair scheduling means you get great latency even with the bandwidth completely maxed out. It actually gets split quite evenly between not just connections but groups of connections from the same hosts. Works best if you accept more context switches via net.ipv4.tcp_notsent_lowat.
1
2
Without net.ipv4.tcp_notsent_lowat, nginx will be filling enormous kernel buffers until it blocks. It's nice being able to coerce it into not having so much tunnel vision.
Setting it low does require the application is fast enough to keep buffers filled despite switching often.
Replying to
Out of interest, how low do you have to set it in order to get more fair workloads, and how many active connections is that generally over?
1
Replying to
CAKE with a proper bandwidth limit turning the server or router where it's deployed into the bottleneck is enough to provide very fair scheduling across hosts / connections at the kernel level. Issue is that the send buffers are still enormous so nginx, etc. get tunnel vision.
1
Show replies