Use USB to synchronize large directory between two disconnected servers

Question

I'd like to synchronize a large directory between two disconnected servers with a USB device that's too small to hold the entire directory. Each server has a majority of the data and the changed files are small enough to fit on my USB device.

Is there a tool I can use to do the following steps?

1. Device B: Compute and store hashes from directory D to the usb device
2. Device A: Compute and compare hashes. Add unsynchronized files to the usb device and make note of the deleted files.
3. Device B: Sync changes from usb device to D, now having the same version as Device A.

To my knowledge, tools like rsync, rdiff-backup, and unison don't support this, so my best solution to this would be to write a few simple scripts to do this.

Answer 1

The simplest way would be to get a cross-over cable and connect the two machines. :) But if you absolutely must do it with a flash memory and some duct tape, you could do it like this:

On the destination machine:

• find /some/dir -type f -print0 | xargs -0 sha1sum -b | sort >dst.txt
• copy the dst.txt file to the source machine

On the source machine:

• find /some/dir -type f -print0 | xargs -0 sha1sum -b | sort >src.txt
• comm -23 src.txt dst.txt | cut -b 43- | (cd /some/dir; tar cJvf - --files-from -) | split -b 3500MB -d
• copy the x* files one by one to the destination machine
• rm -f x* src.txt dst.txt

On the destination machine again:

• cat $( ls -1 x* | sort ) | (cd /some/dir; tar xJvpf -)
• rm -f x*

How is this supposed to work:

• the dst.txt and src.txt files contain SHA-1 sums of the files on the two machines
• comm -23 src.txt dst.txt selects the files that either exist only on the source machine, or exist on both but are different
• cut -b 43- keeps only the filenames; this is the list of files that need to be copied
• cd /some/dir changes CWD to the source directory
• tar cJvpf - --files-from - makes an archive of the bunch; --files-from - reads the list of files to archive from stdin, f - tells tar to write the archive to stdout, J means to compress the archive with xz, and v makes tar print the list of files it archives to stderr
• split -b 3500MB -d splits the archive in chunks of 3500MB (I'm assuming you have a 4GB flash memory, and the data won't fit all in a single run; you might not need to do this at all)
• cat $( ls -1 x* | sort ) concatenates the x* files; sort is there to make sure it does that in the right order
• cd /some/dir changes CWD to the destination directory
• tar xJvpf - saves the files; J means decompress with xz, p means restore permissions, v makes the operation verbose, and f - tells tar to read the archive from stdin.

Finally, the @Stéphane Chazelas disclaimer: this assumes that (1) filenames on either machine don't contain newlines, and (2) the directories to mirror are not too deep (there's a limit to the maximum path length of files in tar archives, and it isn't very high - 200 characters IIRC).

Answer 2

Can tar in combination with the -N | --newer | --after parameter help you?. It allows you to backup all the files which are modified since the specified date.

If you create a file list (using find for example) before each run, you can compare the files to see which files are removed. Using that output you can remove the extra files from the destination server.