How to let rsync additionally store an differential changelog in another directory?

Question

I'm using rsync --link-dir to obtain daily snapshots with little space usage due to the hardlinking. It would however be nice to also have a directory that, while mirroring the original structure, only contains (hardlinks to the full backup of) the files that have actually changed. As an example, here's the directory structure before:

source/        <- contains the original data
backup/current <- symlink to the most recent snapshot
backup/12102?  <- recent daily snapshots (irrelevant)

and here the desired structure after:

source/
backup/current <- symlink to the _updated_ snapshot
backup/previous<- symlink to the snapshot that was was "current" before
backup/12102?  <- ...
backup/current.changes  <- (symlink to) directory containing _only_ the files that
                           have changed between "current" and "previous"

Answer 1

Here's the script I ended up using:

#!/bin/bash
set +x
SRC=$1
BCK=$SRC/.snapshots
CUR=$BCK/current
DAT=$(date +%Y-%m/%y%m%d)
DST=$BCK/$DAT
PAR="-aP --chmod=a-w,o-rwx --no-owner"

LNK="--link-dest=$CUR --link-dest=$DST.changeset"
CMP="--compare-dest=$CUR --prune-empty-dirs"
XCL="--exclude-from $BCK/.rsyncignore"
LOG="$BCK/log/$DAT"
LGP="--log-file $LOG"

mkdir -pm750 $DST.incomplete
mkdir -pm750 $DST.changeset.incomplete
mkdir -pm750 $(dirname $LOG)

rsync $PAR $CMP $XCL $LGP.change.log $SRC/ $DST.changeset.incomplete | tee $LOG.change.out
mv $DST.changeset.incomplete $DST.changeset
find $DST.changeset -type d -empty -delete
rsync $PAR $LNK $XCL $LGP.log $SRC/ $DST.incomplete | tee $LOG.out
mv $DST.incomplete $DST
rm -f $CUR
ln -s $DAT $CUR

It will create snapshots in a DST=$1/.snapshots/YYYY-MM/yymmdd fashion, and $DST.changeset will contain only the non-empty directories containing files that have been created or modified. Deletion (and indirectly mv) is not recognized, although a third rsync swapping $DST and $CUR would create the reverse-modlog, which then could be merged with the incomplete changeset maybe using diff syntax for some intuitive renaming... Or, you just parse rsync's log output or directly use git since you're basically versioning...

---

old answer:

My first thought was using a hardlink copy of the previous backup together with the --backup-dir parameter, but that would obtain the previous version of the files that have changed, which is the oposite of what I am looking for. The trick is inverting this:

• First, create the usual snapshot using hardlinks:
  rsync -a --link-dest=/backup/current /source/ /backup/$TODAY.incomplete
  (You may have to use --no-owner --chmod=... etc. to make rsync definitely use hardlinks)
• Now overwrite the new snapshot with the previous one, but let --backup move the changed files:
  

  rsync -ab --backup-dir=/backup/$TODAY.changelog.incomplete --delete /backup/current/ /backup/$TODAY.incomplete
  mv /backup/$TODAY.changelog.incomplete /backup/$TODAY.changelog

• Congratulations, you screwed up the new snapshot at the expense of creating a changelog!
  Fix this (and cleanup) with
  

  cp -alf /backup/$TODAY.changelog/* /backup/$TODAY.incomplete
  mv /backup/$TODAY.incomplete /backup/$TODAY
  mv -f /backup/current /backup/previous
  ln -sf /backup/$TODAY /backup/current

Note that this "changelog" actually only contains changed or new files (the latter only if you didn't forget the --delete); neither deletions nor mvs are tracked. Some modifications may fix this...