Pushshift will be using Zstandard compression going forward. I am going to write a "cheat sheet" for all the major OS flavors on how to use the zstd tool to successfully decompress .zst files.
The reason for using Zstandard is primarily for its excellent decompression speeds.
Conversation
Also, Zstandard compression offers:
- High compression ratios (when using level 22 combined with long windows)
- Extremely fast decompression (much better than zlib, bz2, etc.)
- Strong open-source community support
- Ease of use
Replying to
Hi Jason, thank you very much for your work! We really appreciate it in our lab. I'm using the (latest version) of the zstd tool and can't decompress the newest Reddit monthly files. I get "not in zstd format". Am I doing something wrong?
1
Show replies