Conversation

Turns out there is an OS-wide limit on how many threads you can have, and if you create an *operating system thread* for each host on your network (???) you will hit the limit

Quote Tweet

Nick Craver

@Nick_Craver

Nov 28, 2020

Just found the AWS US-East-1 outage postmortem is up. Overall: increasing capacity causing more syncing overhead then slamming against configuration limits was the start of the cascading failure - an interesting read! Well worth a few minutes: aws.amazon.com/message/11201/

Replying to

yeah, the max pid value

Replying to

and

Linux vm.max_map_count also acts a per-process thread limit and is only 65530 by default even though it can be drastically higher. Stack overflow is generally detected via a guard region, and if you want to handle those errors you need to set up an alternate signal stack too.

12:51 AM · Nov 29, 2020Twitter Web App

Replying to

and

Each thread requires up to 2 mappings or 4 mappings when supporting catching stack overflows. It can be higher for a hardened implementation. Since adjacent, identical mappings get merged, the count per thread will be lower until the address space gets fragmented over time.

Replying to

and

Hardened runtimes often require additional mappings per stack. There are also a lot of other memory mappings beyond thread stacks. The 65530 limit isn't very reasonable for 64-bit. It doesn't even support common cases if you use a hardened allocator heavily using guard regions.