Conversation

Daniel Micay

@DanielMicay

Latest release of nginx mainline (1.21.6) works around the terrible load balancing of Linux's EPOLLEXCLUSIVE: https://nginx.org/en/CHANGES https://trac.nginx.org/nginx/ticket/2285… https://github.com/nginx/nginx/commit/96c342e56035a9676180d03b4659d5b05b9c6b07… Sad that this is still a problem for Linux years after the kernel feature was added in 2016.

github.com

Events: fixed balancing between workers with EPOLLEXCLUSIVE. · nginx/nginx@96c342e

Linux with EPOLLEXCLUSIVE usually notifies only the process which was first to add the listening socket to the epoll instance. As a result most of the connections are handled by the first worker p...

9:31 PM · Feb 20, 2022·Twitter Web App

2

Retweets

8

Likes

Daniel Micay

@DanielMicay

·

Feb 20

Replying to

@DanielMicay

They've been using EPOLLEXCLUSIVE since shortly after it became available. It replaced needing accept_mutex to avoid unnecessary wakeups. Linux could have used round-robin load balancing for EPOLLEXCLUSIVE as they do for REUSEPORT instead of pathological behavior being standard.

1

3

Daniel Micay

@DanielMicay

·

Feb 20

REUSEPORT isn't a strict upgrade over EPOLLEXCLUSIVE. It distributes the connections (TCP) / messages (UDP) via round-robin without needing synchronization across them. EPOLLEXCLUSIVE avoids giving work to a worker that's currently not waiting but isn't as ridiculously scalable.

1

3

Daniel Micay

@DanielMicay

·

Feb 20

It would help if nginx finally got upstream support for async open. REUSEPORT will work increasingly well as the remaining blocking goes away. They still don't use io_uring but mostly because thread pool AIO works well enough that io_uring would mostly just reduce CPU use a bit.

4