Huh- I knew rsync was slower for local files than cp, but didn't realize by how much. It's around 5x for my current workload!
Conversation
Replying to
Can pass --inplace to have rsync write directly to the destination files rather than using temporary files and renaming them to the destination name after the transfer for that file is finished. Massively reduces the amount of data written to storage for updating existing files.
1
3
It can increase the amount of data it needs to transfer over a network due to how the delta transfer loses the ability to reuse the already overwritten data. It defaults to --whole-file for non-network operations anyway so it's not using deltas and there's no loss of performance.
1
2
None of the traditional tools are NVMe and multi-core aware. It would be a lot faster to have a bunch of read and write operations happening in parallel. rsync also has fairly strict phases and doesn't know how to do the vast majority of it in parallel.
Quote Tweet
Replying to @encthenet
It also does the entire thing in advance and it entirely blocks progress on uploading while calculating all the checksums. It's very impractical for syncing incremental changes to a very large overall amount of data. It's a lot more than just not being parallel / async.
1
1
Hate waiting around waiting for it to hash all files on both ends with a single thread in the -c mode before it even starts transferring. It pretty much needs to be rewritten from the ground up to actually properly use threads/AIO and to have the phases able to run in parallel.
If you have a way to divide up the work, the parallel tool is nice:
github.com/GrapheneOS/gra
It's hard to use that for general-purpose rsync use but it's fairly trivial if you always have the same files or at least the same top-level directories. Run a separate rsync for each.
1