1

I have following directory structure on remote machine

 ALLLibs/LibA/LibA.1
 ALLLibs/LibA/LibA.2
 ALLLibs/LibB/LibB.1
 ALLLibs/LibB/LibB.4

for info LibA.1 and LibA.2 are version of a directory, content of this two folder may be almost same and there can be more such folders.

I want to sync AllLibs to my other machine where network is too slow. and Only Available option is via network

I know rsync is best for syncing files but I believe it does 1-1 mapping of files and sync them.

One use case I have is ALLLibs/LibA/LibA.1 if is synced on my other machine already and later on ALLLibs/LibA/LibA.2 gets created on remote, I want the sync to be happened via patch of the previous lib.

  • If I understand correctly, you want the sync to be clever enough to detect similar binary files and only transmit the difference? rsync cannot do this I believe, since it works on file-by-file basis. However, incremental compressed backups may be exactly what you want: They create a compressed difference of a previous snapshot. You could automate the creation, transmission and extraction of such an archive. – Sebastian Aug 28 '15 at 6:51
0

Make sure you are using rsync -z to compress over the network. Other options to consider are -H (hardlinks) and -S (sparse files).

Before you run your rsync, do a version with -nv which does no updates but shows you which files will be updated.

Out of the list of files, use your knowledge to match where a missing file might already exist on the remote, and then login to the remote and copy these files first.

For example, -nv might show updates needed for:

ALLLibs/LibA/LibA.2
ALLLibs/LibA/LibA.2/file1

but if you know it is probably a copy of ../LibA.1/file1, then write a script to create and populate LibA.2 from LibA.1, and run it on the remote. It can be as simple as cp -rp. When you do the real rsync, similar files will already be there, and changed files will be updated. You will need --delete to remove files that should not be there (be careful with this option: test with -n first).

Be careful in your copy command on the remote: copying a file usually sets a new timestamp on the new file. rsync by default compares size and timestamp. If they differ, the file will be updated. Use rsync option -c to instead checksum each file to see if it differs.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.