Pushshift will be using Zstandard compression going forward. I am going to write a "cheat sheet" for all the major OS flavors on how to use the zstd tool to successfully decompress .zst files.
The reason for using Zstandard is primarily for its excellent decompression speeds.
Conversation
Replying to
Also, Zstandard compression offers:
- High compression ratios (when using level 22 combined with long windows)
- Extremely fast decompression (much better than zlib, bz2, etc.)
- Strong open-source community support
- Ease of use
1
2