Conversation

🤣🐧🤡 This would not even be an issue if everyone hadn't jumped on the epoll 🤡🚗

Quote Tweet

May 1, 2021

I'm not surprised to find out that Linux kernel socket load balancing is in a terrible state with no good options. Some background: blog.cloudflare.com/the-sad-state- Traditional epoll wakes every thread and they race to accept the connection. EPOLLEXCLUSIVE fixes it but uses LIFO order.

Show this thread

Daniel Micay

@DanielMicay

Replying to

@RichFelker

The Linux kernel didn't really provide other viable options. The least bad option not requiring patching the kernel is probably REUSEPORT combined with dealing with the BPF nonsense they expect you to do. You still use epoll but with a socket per worker instead of sharing one.

12:20 AM · May 2, 2021Twitter Web App

Replying to

Just all workers blocked in accept() for a single socket works fine. Only one of them wakes, the one that gets the connection.

Replying to

Blocking accept in Linux has some serious issues itself and the way it's implemented doesn't scale. You have to use edge-triggered epoll to avoid O(n) algorithms. They don't really give you much of a choice. It's why nginx used to do this in userspace and probably still should.

Show replies

Replying to

and

Blocking accept uses FIFO rather than LIFO, but has other issues. It requires a weird non-standard approach to properly handle graceful reloads, etc. It doesn't really make any sense that EPOLLEXCLUSIVE uses a different order than blocking accept by default. It's a weird choice.

Kumar Kartikeya Dwivedi

Replying to

and

IIRC I think it had something to do with LIFO favoring workers that were "cache hot", but then my guess is that also means those threads get more wakeups and overall concurrency is limited as a result.

Show replies