Ah - I've been using systemd-oomd (Fedora 34 defaults) - it triggers on PSI and targets the app causing the stalls (IIRC also avoids targeting shell) - I can successfully report no more shell/X kills and no freezes for months!
Overengineered? probably. Seems to work though. 😅
Conversation
Replying to
disabling overcommit disables the OOM killer. the memory hog will crash on its own if it doesn't handle malloc() returning NULL correctly.
1
so what you want to do is monitor for PSI stalls, and add additional swap as needed. but if a given process is hogging ram, stop adding swap just for that process.
1
Replying to
got it - I've just seen enough "clever" use of overcommit - for example, mmap'ing 32G of PROT_NONE memory between heaps to avoid heap overruns from succeeding in JSCore - phakeobj.netlify.app/posts/gigacage/, or using CoW+fork to save in Redis - redis.io/topics/faq#bac that...
3
Replying to
as far as I know, PROT_NONE mappings do not count against the commit charge. i also do not see how redis's CoW+fork is affected by overcommit
1
Replying to
FAQ says with overcommit off, if you have a 3G dataset and 2G free mem, then fork() fails since w/o overcommit Linux tries to guarantee that the child can write to every page, and there's isn't enough free mem for it. w/ overcommit, Linux promises away mem that's never used
1
Replying to
yeah, my advice for that situation is: don't run a production service under memory pressure
1
Replying to
<50% free mem on a Redis box isn't mem pressure unless you rewrite your entire db in the time it takes to save to disk; which I haven't seen on prod
running against the barrier is a major cost save as long as you can supply the needed disk IOPS
2
1
if overcommit is off, does that mean that 1000 threads allocate 8GB of memory with the default stack allocation?
1
Linux main thread stack is special. It dynamically grows the virtual memory mapping as the stack is used.
If you want a guarantee that you'll be able to use the memory with overcommit disabled, you need to manually map the amount of stack you intend to use in the main thread.
1
2
POSIX thread stacks are normally implemented via mapping the the maximum amount of usable stack space with a guard page to catch stack exhaustion. It's not the only approach. It's entirely possible to mimic the default dynamic growth of the main thread stack or do other things.
it's possible yeah, to dynamically expand to the virtual allocation. but if you take a normal linux userspace and turn off overcommit, you're not going to get that right. it's just going to blat out a big 8MB mapping, which will need to be mapped. unless libc's handle this case?
1
The main thread stack on Linux is not a giant mapping with a bunch of space reserved unless you go out of the way to either touch all of the memory or map it.
Probably do want to do that to avoid stack exhaustion from memory exhaustion for systems designed to avoid overcommit.
1
1
Show replies