3

What I want:

I'm trying to back up many files to S3 each day. I want to compress these into archives daily so that each individual backup doesn't cost as much. Therefore, I want a backup system whereby I can create differential archives from the previous day to be backed up to S3.

My current reasoning is to create an initial backup, and then base backups per day that are differential wrt that initial backup.


Example:

Say I have a directory named dir containing a file, a.txt:

dir:
  a.txt

Then let's say I run rsync -r dir backup.1, or some other backup program, such as, but not limited to, rdiff-backup. Great stuff, I have a backup located at backup.1.

Now, I add one file, b.txt, so that folder's contents is as follows:

dir:
  a.txt
  b.txt

Is there an rsync, or another backup utility, command I can run that will give me a separate directory, backup.2, that contains only the file b.txt?

If I modify the permissions of a.txt, will that file also be included?

What I've tried:

I've tried rsync and rdiff-backup, with a range of flags, but I cannot get anywhere. I've been doing it for hours now, and I cannot figure it out.


EDIT:

Halfway there!

rsync -n -rpgov backup/ backup-copy/

Please note the -n means this does a dry-run,, so you can happily test this without wrecking your own local directories with file changes.

This outputs the difference between two local folders. However, I cannot for the life of me figure out how to get this output, through rsync's options, to a new directory.

I've tried using rsync's --compare-dest flag:

rsync -n -rpgov --compare-dest=backup/ backup-copy/

but this seems to output different files to the above command.How to push this over the edge?

  • Are the leading dashes part of the filenames, e.g. - a.txt ? – roaima Sep 4 '18 at 21:07
  • Is your rationale for having backup trees of changed files so that you save disk space? (If so, what's wrong with rsnapshot and its hard linked files?) How would your partial backup tree represent a deleted file or directory? – roaima Sep 4 '18 at 21:09
  • 1
    @roaima It isn't for saving file space. I create a lot of individual files (in the hundreds of thousands) that need to be backed up. We get charged in S3 per GET/PUT request, and it's becoming prohibitively expensive. Having one archive makes the process significantly cheaper – Nick Bull Sep 4 '18 at 22:03
  • @roaima The leading dashes are not part of the filename – Nick Bull Sep 4 '18 at 22:04
  • 1
    Thanks. Your comments about S3 costs would be really useful in your question as background information. It's the sort of information that can help direct someone to a most useful suggestion. – roaima Sep 4 '18 at 22:05
3

Here's the answer!

rsync -rpgov --compare-dest=/home/nick/docs/test/backup-copy backup/ backup-output/

Note the complete path, and the lack of trailing slash at the end of the --compare-dest flag. This made the difference!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.