how to get rsync to make hard links to the source dir, while also backing up changed files?

Question

I've got two backup directories that live on the same filesystem on my backup server. The first is called "clone" - it contains a clone of my laptop that is remotely updated nightly via rsync. The second is called "backup", which is a weekly rsync snapshot of only the important parts of clone. To save space, "backup" is created as hard links to clone instead of copies, using --link-dest:

rsync -avum --link-dest=/clone /clone/ /backup

Now I want to also use the --backup option to copy the old versions of changed files from backup to a holding area, in case I need them or accidentally delete something important. This works fine without --link-dest:

rsync -avumb --backup-dir=/holding/2016_10_22 /clone/ /backup

However, this creates copies of the changed files in backup, wasting space - I want hard links. But if I add the --link-dest parm back in:

rsync -avumb --backup-dir=/holding/2016_10_22 --link-dest=/clone /clone/ /backup

...then only deleted files are backed up. Changed files are silently hard linked. The reason (I believe) is that --link-dest shares the logic of --copy-dest. I.e., if the source file is unchanged relative to the copy-dest (or link-dest) file, then it is not transferred, but instead copied/linked from the copy/link-dest dir to the target dir. Because I'm using the source dir as the link-dest dir, all non-deleted files are "unchanged", and handled silently.

I could do this in two steps: first --backup without --link-dest, then again --link-dest without --backup. (Newer versions of rsync will replace identical files with hard links.) But I'd really prefer to do it all at once.

Is there a way of doing --backup while only creating hard links? (Really what I want is "regular" rsync with hard linking instead of file transfer. My use of --link-dest seems like a bit of a hack, given the intended logic of that option.)

Bonus question: the man page seems to indicate that using --link-dest only on empty targets is preferred:

This option works best when copying into an empty destination hierarchy, as existing files may get their attributes tweaked, and that can affect alternate destination files via hard-links. Also, itemizing of changes can get a bit muddled.

The bit about itemizing getting "muddled" is a bit vague. Is using --link-dest on a non-empty target really "dangerous", assuming I don't care too much about file attributes? Can anyone give an example?

Answer 1

As mentioned above, I ended up running the process in two steps.

src = the "live" clone directory, a mirror of my laptop = primary backup
dst = the weekly snapshot of important parts clone
trashdir = the items from src that don't exist in dst, because they have since been deleted, sorted into date-stamped directories

cmd = ("rsync -avumb --stats --delete --delete-excluded --filter='merge %s' --backup-dir=%s %s %s" %
       (filterfile, trashdir, src, dst))

cmd = ("rsync -avum --stats --delete --delete-excluded --filter='merge %s' --link-dest=%s %s %s" %
       (filterfile, src, src, dst))

The first step makes a backup of the important parts of clone, and saves the files that have been deleted since the last backup, to trashdir. (There are reasons why I want this intermediate backup of only selected files, and only want it updated weekly.)

The second step converts the files in backup to hard links pointing at clone. The result is that backup takes zero actual space. trashdir is by definition not hard-linked files, because it consists of only files that were deleted from clone & backup.

I'm not entirely sure whether the --delete-excluded flags are necessary (particularly in the second command). I've left them in there, in case I change filterfile, which defines which parts of clone are to be ignored when creating backup.

I've found that in five years, trashdir has grown to approximately the size of clone, so total size = clone x2, which is acceptable to me, given that I have six years of history for deleted files, and can easily prune them by date.

In addition to the above, I have a script that runs cp -al to copy clone into about a month of date-stamped rotating hardlinked snapshots. This covers me for files that have been changed, rather than deleted. Total size for a month seems to be about half the size of clone.

So total disk space is ~2.5 of the clone, and I have:

• the clone itself, updated nightly
• a backup of select files which is stable for a week
• a month's worth of versioned clone snapshots
• six year's worth of deleted files

Which I feel is pretty good protection against loss of the original disk, overwriting a file and needing an older version, and deleting a file and needing it later.

It's a bit convoluted, and probably could have been achieved with third-party software, but it works for me and is built on low-level tools that aren't likely to disappear or significantly change functionality.

(@gsl - Actually, thanks for requesting the update. I discovered that one of my scripts broke when I updated to python3, and hadn't been running for a couple of months. I need to pay more attention to my error logs!)

Definitely still interested in ways to streamline this, though - so feel free to comment if the things I'm doing could be accomplished in some easier way.