Conversation

Latest release of nginx mainline (1.21.6) works around the terrible load balancing of Linux's EPOLLEXCLUSIVE: nginx.org/en/CHANGES trac.nginx.org/nginx/ticket/2 github.com/nginx/nginx/co Sad that this is still a problem for Linux years after the kernel feature was added in 2016.

github.com

Events: fixed balancing between workers with EPOLLEXCLUSIVE. · nginx/nginx@96c342e

Linux with EPOLLEXCLUSIVE usually notifies only the process which was first to add the listening socket to the epoll instance. As a result most of the connections are handled by the first worker p...

Daniel Micay

@DanielMicay

They've been using EPOLLEXCLUSIVE since shortly after it became available. It replaced needing accept_mutex to avoid unnecessary wakeups. Linux could have used round-robin load balancing for EPOLLEXCLUSIVE as they do for REUSEPORT instead of pathological behavior being standard.

9:45 PM · Feb 20, 2022Twitter Web App

Replying to

REUSEPORT isn't a strict upgrade over EPOLLEXCLUSIVE. It distributes the connections (TCP) / messages (UDP) via round-robin without needing synchronization across them. EPOLLEXCLUSIVE avoids giving work to a worker that's currently not waiting but isn't as ridiculously scalable.

Daniel Micay

@DanielMicay

Feb 20

It would help if nginx finally got upstream support for async open. REUSEPORT will work increasingly well as the remaining blocking goes away. They still don't use io_uring but mostly because thread pool AIO works well enough that io_uring would mostly just reduce CPU use a bit.