Are there any open source services for managing zstd dictionaries? ie: training, fetching, versioning, testing, etc?
Conversation
Replying to
Lack of decent tooling and standards is why there's still no usage for HTTP compression. Brotli having a standard dictionary that's used by default means almost everyone is using that. zstd not having one means hardly anyone bothers doing it since you need to generate them.
1
I don't think there's really a better way of dealing with it than scripting it yourself and managing the inputs for generating dictionaries in git where you need to handle versioning, etc. yourself. We might switch to zstd from DEFLATE for Auditor QR codes at some point.
1
Considered using github.com/vkrasnov/dicta to replace simply concatenating a few samples together for a pre-shared DEFLATE dictionary but it's too much trouble to get a bunch of samples and then update them all and generate a new dictionary. The dumb approach worked too well.
1
Problem we ran into with zstd is that the pure Java ports don't support the pre-shared dictionary system. It's nice that you can generate them with zstd but we didn't want to use the native library with some JNI wrapper. Also haven't even confirmed it works better for this yet.
Replying to
In my case I basically want to be able to look at a sample of data, deploy the dictionary, and then have services pull the latest dictionary and tag the data they're compressing with the version used.
But all of the 'deploy, grab, train' bits seem to be up to me to write
1
Which is fine, I'll just write a service that provides that kinda API and manages the dicts, but I thought maybe someone had done it.