I'm not surprised to find out that Linux kernel socket load balancing is in a terrible state with no good options. Some background:
blog.cloudflare.com/the-sad-state-
Traditional epoll wakes every thread and they race to accept the connection. EPOLLEXCLUSIVE fixes it but uses LIFO order.
Conversation
LIFO order is terrible for a web server. HTTP connections are generally long-lived and reused for mixed / varying workloads. That's even more true with HTTP/2 where clients are only supposed to make a single connection to each server and multiplex everything over it concurrently.
1
6
In a standard nginx setup on Linux, it uses EPOLLEXCLUSIVE. This gives nearly all the connections to the same worker until it starts getting overloaded. Even then, the most overloaded workers still keep getting the most connections among handling other events. It's pretty awful.
Replying to
The alternative that's offered is using reuseport to have the kernel evenly distribute new connections across workers.
It doesn't account for the varying usage of connections. It happily hands out new connections to a worker that's not even idle. It's not as nice as it sounds.
1
1
The kernel developers were gradually improving REUSEPORT in different ways. However, they stopped and allowed some of it to regress. You're supposed to use BPF to give the kernel a load balancing approach specific to the application. Of course, applications don't actually do it.
2
1
2
A decent approach for many applications would be if epoll used FIFO instead of LIFO order. It would work like reuseport but only distributing connections to idle workers. It was proposed in 2015 with EPOLLEXCLUSIVE but didn't land. It's apparently what Cloudflare uses themselves.
1
3
marc.info/?l=linux-fsdev was an attempt at reviving it. Cloudflare tends to not be persistent enough to get through the hassle of getting changes landed in the Linux kernel or nginx.
They deal with the initial technical aspect but not the politics / evangelism to get it landed.
1
4
They also tend not to invest resources into making things into a more generic solution suitable beyond their use case. Makes sense for them. They clearly don't see that much value in their changes being upstream. The upstream projects tend not to care much about latency, etc.
2
3