3

At home I have the following backup scheme: All machines daily rsync to the servers $FULL_COPY directory. The server takes snapshots of that directory daily (also weekly/monthly), keeping at most X snapshots. Also using rsync:

rsync -ar --link-dest=$LATEST_SNAPSHOT $FULL_COPY $NEW_SNAPSHOT

This works great, files not changed since last snapshot get hard linked. The problem is large files (to be precise Thunderbird's inbox), a small change means an entire copy of the file. My snapshots are easily a couple of GB, but maybe only a few MB has changed.

Is there a smarter way to do this? (my $FULL_COPY is also replicated on a remote machine, so I'm not in need of this redundancy). My only idea is a CoW filesystem supporting snapshots. But that would mean major surgery on my system which I'm not fond of.

  • 1
    There is also rsync's --write-batch ... – derobert Oct 24 '13 at 20:13
  • Interesting, but I don't see how that is applicable to my problem (where I want multiple snapshots). – stacker Oct 24 '13 at 21:13
  • 1
    Instead of snapshots, you store the result of --write-batch/--only-write-batch. You can then apply those (--read-batch) to construct any of the snapshots. You only make snapshots twice per month, for example, instead of daily (and base your batch on the most-recent snapshot). – derobert Oct 24 '13 at 21:19
1

I give kind of a long write-up here but it is really pretty simple, use patch.

Make the $FULL_COPY as usual, but for the $NEW_SNAPSHOT, exclude the very large file(s).

For now, let's say it is one file and the path to the file is /home/myuser/.mail/INBOX you would run:

rsync -ar --link-dest=$LATEST_SNAPSHOT $FULL_COPY $NEW_SNAPSHOT --exclude=/home/myuser/.mail/INBOX

This will decrease the time to take the snap-shots dramatically.

Next, let's fill out the paths for more clarity. Let's say the 'root' directory of $FULL_COPY is under /mnt/full, so the full backup of the user's mail directory goes to /mnt/full/home/myuser/.mail/INBOX. Similarly, snap-shots similarly go under /mnt/snap-mm-dd-yy.

Then to get only a delta of the inbox and put it in the correct dated snapshot at a location that makes sense, run:

diff -u /mnt/full/home/myuser/.mail/INBOX /home/myuser/.mail/INBOX > /mnt/snap-mm-dd-yy/home/myuser/.mail/INBOX.delta

What you have done here is use diff to create a patch file of today's mailbox file for the full backup mail file (the -u option of diff does that) and send the patch file to the snapshot directory under a predictable location (i.e. the user's .mail directory for that date).

Finally, if/when you need to restore the mailbox back to the way it looked on the day of the snapshot because you lost the original, first restore everything to the original path using your $FULL_COPY and $SNAP_SHOT. Then, run the patch program to recover the INBOX delta:

patch < /mnt/snap-mm-dd-yy/home/myuser/.mail/INBOX.delta

That's it.

Notes:

1) If you look at the INBOX.delta, the first two lines are the paths to the old and new files, so you could edit this if you didn't want to patch using the original path(s).

2) Though the delta grows a little larger each day, it still remains much smaller than the whole. Growing the delta until the next full is going to be a more efficient use of resources than creating multiple patches.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.