I just got a ~2x throughput increase in a timely dataflow computation by ... pre-allocating 8GB in each worker thread. Do I do anything with this memory? Nope. I just allocate it. Apparently it stops the memory subsystem from madvising all freed memory immediately. Yay systems.
-
-
Replying to @frankmcsherry
Is that jemalloc or system allocator or ... that is doing that?
1 reply 0 retweets 0 likes -
Replying to @ArmyOfBruce
Jemalloc, it seems. The system allocator (on Linux, at least) seems to do a fine job on this problem and not madvise memory I'm about to need again in just a moment. I feel bad because in the OSDI submit we blamed the "linux memory subsystem", but it seems they were just fine...
1 reply 0 retweets 1 like -
Replying to @frankmcsherry @ArmyOfBruce
Jemalloc is still the default
@rustlang allocator (for binaries). I'm glad they recently made it stable to swap it out! :D2 replies 0 retweets 1 like
Replying to @frankmcsherry @ArmyOfBruce
Only on some platforms; Windows doesn’t use it for example
7:47 AM - 9 Sep 2018
0 replies
0 retweets
0 likes
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.