I have a bunch of very large "almost-append-only" files (meaning that a few bytes at the end are changed every day).

I know of the -B, --block-size=BLOCKSIZE to check deltas in blocks.

Is it possible to skip the first N blocks (or start checking from the block at 99%)?


You cannot specify the starting block number, but as per the original text of your question, if you can guarantee the source files will only ever be extended and not overwritten, you can use the --append option.

From the documentation (man rsync),

--append This special copy mode only works to efficiently update files that are known to be growing larger where any existing content on the receiving side is also known to be the same as the content on the sender. The use of --append can be dangerous if you aren't 100% sure that all the files in the transfer are shared, growing files. You should thus use filter rules to ensure that you weed out any files that do not fit this criteria.

You must use rsync without --append if "a few bytes at the end are changed". If you're copying across a network (rsync -a path/to/source remoteHost:path/to/destination) then you will get a full file read of source and destination each time you run rsync, but only a checksum per block, and the blocks that have changed, will actually be transferred across the network. Unfortunately, if you're doing a local copy (rsync -a path/to/source path/to/destination) then the entire file will be copied if its size has changed.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.