I just compressed one hour of verified Twitter data using these Zstandard options:
zstd -v -22 --ultra --long TW_verified_2019-11-28-22 --zstd=wlog=30,hlog=26,clog=28,slog=26,mml=7,tlen=999,ovlog=9,lhlog=26,lblog=8,lmml=4096
Compression Ratio: 5.62% (337174007=> 18940872 bytes
Conversation
Replying to
5.62% of the original size is the best compression level I've seen yet -- it's quite amazing.
1
2
Replying to
If all those options seem confusing, running "man zstd" gives great documentation on what each parameter does. Mainly they adjust window size, match length matching and hash table settings.
1
Replying to
. -- you might be interested in this. I think I remember you were testing different compression schemes.
4
Replying to
zstd is awesome especially the improvements in 1.4.4 - my benchmarks at twitter.com/centminmod/sta 😎
Quote Tweet
Compression algorithm benchmarks zstd 1.4.4 vs brotli vs gzip vs pigz vs pbzip2 vs pxz community.centminmod.com/threads/18669/ #compression #zstd #brotli #gzip #xz #bzip2 #lz4 #sysadmin #devops #linux #webpef #linux #pigz
2
1
Replying to
Yes! I have been very impressed with zstandard and each new version seems to improve compression ratios.
I'm really hoping that ZFS can incorporate zstandard compression sometime around the release of Ubuntu 20.04 LTS.
1