Conversation

Turns out there is an OS-wide limit on how many threads you can have, and if you create an *operating system thread* for each host on your network (???) you will hit the limit

Quote Tweet

Nick Craver

@Nick_Craver

Nov 28, 2020

Just found the AWS US-East-1 outage postmortem is up. Overall: increasing capacity causing more syncing overhead then slamming against configuration limits was the start of the cascading failure - an interesting read! Well worth a few minutes: aws.amazon.com/message/11201/

Replying to

yeah, the max pid value

Replying to

and

Linux vm.max_map_count also acts a per-process thread limit and is only 65530 by default even though it can be drastically higher. Stack overflow is generally detected via a guard region, and if you want to handle those errors you need to set up an alternate signal stack too.

Replying to

and

Each thread requires up to 2 mappings or 4 mappings when supporting catching stack overflows. It can be higher for a hardened implementation. Since adjacent, identical mappings get merged, the count per thread will be lower until the address space gets fragmented over time.

12:59 AM · Nov 29, 2020Twitter Web App

Replying to

and

Hardened runtimes often require additional mappings per stack. There are also a lot of other memory mappings beyond thread stacks. The 65530 limit isn't very reasonable for 64-bit. It doesn't even support common cases if you use a hardened allocator heavily using guard regions.