First test with 96 core server using zstd to compress 25,881,600 tweets.
time zstd -qcd tweets.zst | zstd -v -19 --long=31 -T96 > test.zst
5.07% (238960762511 => 12123745253 bytes)
real 34m31.667s
user 1599m22.198s
Conversation
For some reason, zstd wasn't using all possible threads. I'm not sure if this was an I/O situation or due to CPU cache issues. Need to do more testing. It averaged around ~53 load average for the entire compression.
Replying to
It was able to reduce 239 GB of data down to 12.1 GB in under 35 minutes.
1
11
Replying to
Most probably I/O played an important role here. You got ~116MB/s avg., which is sensible for a sustained non-SSD I/O.