First test with 96 core server using zstd to compress 25,881,600 tweets.
time zstd -qcd tweets.zst | zstd -v -19 --long=31 -T96 > test.zst
5.07% (238960762511 => 12123745253 bytes)
real 34m31.667s
user 1599m22.198s
Conversation
Replying to
For some reason, zstd wasn't using all possible threads. I'm not sure if this was an I/O situation or due to CPU cache issues. Need to do more testing. It averaged around ~53 load average for the entire compression.
2
3
It was able to reduce 239 GB of data down to 12.1 GB in under 35 minutes.
1
11
This Tweet was deleted by the Tweet author. Learn more
Show replies
Replying to
Maybe you'd like to give FAPEC a try, just for curiosity?
dapcom.es/get-fapec/
It supports up to 63 threads with large variety of algorithms, incl. LZW, FAPECLZ, tabular CSV-like, etc.
Most probably Zstd wins on text data, though.