grapheneos.org/articles/serve is still working nicely for making the most of our server bandwidth with our greatly expanded infrastructure.
CAKE + net.ipv4.tcp_notsent_lowat gives amazing fair load balancing and latency.
BBRv1 hurts too much and doesn't really seem to provide much...
Conversation
Tuning net.ipv4.tcp_notsent_lowat isn't hard since it doesn't need to be as high under actual load. It's easy to tell it's working from having a lot more context switches since it avoids filling enormous buffers. nginx has no concept of fair scheduling at all so it helps a ton.
1
2
Can set it higher if the server doesn't have CPU cycles to spare but that's pretty much a non-issue for a reverse proxy or static file server.
Might as well be using CAKE and net.ipv4.tcp_notsent_lowat if you're nowhere close to maxing out the CPUs and that's normally the case.
1
3
Fair scheduling means you get great latency even with the bandwidth completely maxed out. It actually gets split quite evenly between not just connections but groups of connections from the same hosts. Works best if you accept more context switches via net.ipv4.tcp_notsent_lowat.
Replying to
Without net.ipv4.tcp_notsent_lowat, nginx will be filling enormous kernel buffers until it blocks. It's nice being able to coerce it into not having so much tunnel vision.
Setting it low does require the application is fast enough to keep buffers filled despite switching often.
1
1