Suppose that we have two replicas A and B of a large data set in locations that don't have a good Internet connection. As a rule of thumb, let's say it's possible to transfer metadata but not actual file data, but a full offline solution would be even better. One of the replicas is a master, the other one is sort of backup that needs to be updated semi regularly.

The way the two data sets are kept in sync is... old school. Someone copies the files that need to be updated to an external drive, physically gets from A to B where the files are updated in the other replica. However, it gets increasingly difficult to keep track of which files need to be transferred, especially in error cases like a corrupted file.

It's impractical to have a full copy of the data set on the external drive as it's at least an order of magnitude larger than its capacity. A typical file size is a few gigabytes and the entire data set shouldn't grow past lower tens of terabytes in foreseeable future. A typical update is a file addition, updates to existing files and deletes are fairly rare.

What matters the most is the bytes being preserved, I don't mind about inconsistent timestamps or file permissions.

One can do this manually storing checksums of files in replica B on the portable drive and checking for mismatches on replica A. But hey, is there an established tool that supports use cases close enough to mine?


  • 1
    Out of interest are you able to comment on the actual size of the data set. These days gigabyte files aren't that scary since we have relatively cheap terabyte hard drives. – Philip Couling Jun 20 at 14:45
  • I updated the question with a few estimates. – Danstahr Jun 20 at 14:47

Take a peek at rsync(1), it is designed for that kind of job.

You might get away using diff(1)/patch(1) (or the xdiff tools) if you know changes are localized.

If you know about the structure of the data, and their changes, you might cobble up some specialized tool. But that is more work, brittle and won't grow with changes.

If it is mostly text files in directories, perhaps even setting it up as a git(1) repository and pulling from there could be a practical solution (it is optimized to fetch and integrate changes efficiently).


Have you taken a look at git patch files? If both location A and location B are git repos, you could simply copy a generated git patch file to the external drive and then apply it to your backup location.

Repo A:

commit 090d683743b24d4fd93cdf2c1fc3fe3613443be0 (HEAD -> master)
Author: somebody <somebody@example.com>
Date:   Thu Jun 20 19:01:14 2019 -0400

    added my love for bananas

commit b58c37b8479ce188c896719336a456d168c56ea9
Author: somebody <somebody@example.com>
Date:   Thu Jun 20 18:57:57 2019 -0400

    initial commit

Repo B:

commit b58c37b8479ce188c896719336a456d168c56ea9 (HEAD -> master, origin/master, origin/HEAD)
Author: somebody <somebody@example.com>
Date:   Thu Jun 20 18:57:57 2019 -0400

    initial commit

Now we can generate a patch file using git.

git diff -p commit_old commit_new > /external_drive/patch.diff

In Repo B:

To see changes:

git apply --stat patch.diff

To apply changes:

git apply patch.diff

Add and commit the changes:

git add -A
git commit -m "added my love for bananas to this directory"

See the new commit:

git log
commit 4122e3deca160a9167fb54456bb688394aa5dcff (HEAD -> master)
Author: somebody <somebody@example.com>
Date:   Thu Jun 20 19:10:19 2019 -0400

    added my love for bananas to this directory

commit b58c37b8479ce188c896719336a456d168c56ea9 (origin/master, origin/HEAD)
Author: somebody <somebody@example.com>
Date:   Thu Jun 20 18:57:57 2019 -0400

    initial commit

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.