If the compression is the bottleneck than parallel implementations of popular algorithms do exist (and tar can use them)
yeah, I was probably wrong -- something wasn't going right, and I think you're right that this wouldn't solve a thing heh.
-
-
tar itself is kinda single-thread-limited due to the need to write the data sequentially. try pigz (for gzip compression) or xz with the `—threads` flag for multi-threaded compression. (tar takes --use-compress-program)
-
> tar itself is kinda single-thread-limited due to the need to write the data sequentially. Interesting! -- Do you by any chance know if it writes sequentially, and then compresses? Or does it do both on the fly? In the case of the former, you could parallel write using offsets
- 2 more replies
New conversation -
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.