do people have a good solution for managing global parallelism on a unix box? the problem is that I want to fork some processes, each of which is going to fork some unpredictable number of processes, and I want global parallelism == number of cores
Conversation
easy solutions that don't work for me include looking at machine load before forking and running some sort of persistent global parallelism manager.
but I guess processes could scan the process table and count instances of themselves before forking? not real elegant...
7
2
ok it looks like POSIX jobserver miiiiight be the way to go here since we're often running under GNU make anyhow. bummer that ninja doesn't support this! gnu.org/software/make/
4
11
Replying to
Make sure you're using Linux 5.6 or later.
git.kernel.org/pub/scm/linux/
It's still fairly high overhead. If your tasks aren't long-lived, you might end up with a significant performance loss from being bottlenecked by the single byte reads and writes to the single global POSIX pipe.