Without net.ipv4.tcp_notsent_lowat, nginx will be filling enormous kernel buffers until it blocks. It's nice being able to coerce it into not having so much tunnel vision.
Setting it low does require the application is fast enough to keep buffers filled despite switching often.
Conversation
Replying to
Out of interest, how low do you have to set it in order to get more fair workloads, and how many active connections is that generally over?
1
Replying to
CAKE with a proper bandwidth limit turning the server or router where it's deployed into the bottleneck is enough to provide very fair scheduling across hosts / connections at the kernel level. Issue is that the send buffers are still enormous so nginx, etc. get tunnel vision.
1
It helps a lot even without net.ipv4.tcp_notsent_lowat and it gradually gets better and better from coercing the applications into switching between connections as you lower it. It's easy to see that lowering it starts causing more context switches + better latency/fairness.
2
You really need net.ipv4.tcp_notsent_lowat on the server itself so you have very low latency feedback to the applications about when the buffers are adequately filled. It just shouldn't be set too low for them to max out the bandwidth which isn't really that hard to tune.
1
For nginx, it's so quick at waking up and pouring data into the kernel buffers with low CPU usage that setting it to 128k works fine for high throughput, at least near 1-2Gbit. It's also only going to bottleneck very fast connections when the server is under low load regardless.
2
Replying to
*nod*, it just needs to be large enough to overcome BDP and scheduler latency
1
1
Replying to
CAKE also adds a decent amount of CPU overhead and so do the extra context switches from much low send buffer limits. Generally, web servers and reverse proxies have a ton of CPU usage to spare so it just doesn't matter. May not want it if you have CPU-bound app server there.
1
It's not really much of an issue on a server though. The overhead is primarily relevant to routers, and is why fq_codel is going to be a lot more broadly deployed than CAKE. CAKE is truly amazing to see in action though when it's actually the bottleneck doing shaping.
1
Also, if you do set net.ipv4.tcp_notsent_lowat somewhat too low, it's not like you've harmed the overall capability of the server under load. Primarily relevant when there's low load and clients have 1Gbit or higher connections + server fails to fill buffer for them fast enough.
1
It's configuring the amount of write buffer size the kernel provides beyond the TCP window size so you aren't ever actually lowering TCP window size. You aren't ever actually lowering the window size by using it. You're just lowering how much it queues up beyond that.
It also reduces memory usage substantially although that isn't really a goal for our usage of it. The kernel is normally extremely aggressive with how much it queues up beyond the optimal window size it decides upon.
1
Replying to
Yeah, it's possible you could gain some cache/locality benefits, but at these scales it's probably lost to noise
1
Show replies