11

One might think that --link-dest'ing to an identical file would work in all cases. But it does not when the file exists, even if the file is out of date/has different contents.

It is because of this, from the rsync man page on --link-dest:

"This option works best when copying into an empty destination hierarchy, as rsync treats existing files as definitive (so rsync never looks in the link-dest dirs when a destination file already exists)"

This means that if y/file exists same as source, and z/file is out of date,

rsync -a --del -link-dest=y source:/file z

will result in TWO inodes (and twice the diskspace) being used, y/file and z/file, which will have the same contents and datestamps.

I came across this because I do daily backups basically with this script run once per day:

mv $somedaysago $today; 
yest=$today; today=`date +%Y%m%d`;
rsync -avPShyH --del --link-dest=../$yest host:/dirs $today

Because my backups span up to 10M files, doing rm -rf $olddir; rsync source:$dir newdir would take way too long (especially when only 0.5% of the files change per day, incurring the deletion and creation of 10M dir entries just to handle 50K new or changed files, which would make my backups not complete in time for the next day).

Here's a demo of the situation:

a is our source, 1 through 4 are our numbered backups:

$ mkdir -p 1 2; echo foo > 1/foobar; cp -lrv 1/* 2
`1/foobar' -> `2/foobar'
$ ls -i1 */foobar
1053003 1/foobar
1053003 2/foobar

$ mkdir a; echo quux > a/foobar
$ mv 1 3; rsync -avPhyH --del --link-dest=../2 a/ 3
sending incremental file list
./
foobar
           5 100%    0.00kB/s    0:00:00 (xfer#1, to-check=0/2)

sent 105 bytes  received 34 bytes  278.00 bytes/sec
total size is 5  speedup is 0.04

$ ls -i1 */foobar
1053003 2/foobar
1053007 3/foobar
1053006 a/foobar

$ mv 2 4; rsync -avPhyH --del --link-dest=../3 a/ 4
sending incremental file list
./
foobar
           5 100%    0.00kB/s    0:00:00 (xfer#1, to-check=0/2)

sent 105 bytes  received 34 bytes  278.00 bytes/sec
total size is 5  speedup is 0.04


$ ls -il1 */foobar
1053007 -rw-r--r-- 1 math math 5 Mar 30 00:57 3/foobar
1053008 -rw-r--r-- 1 math math 5 Mar 30 00:57 4/foobar
1053006 -rw-r--r-- 1 math math 5 Mar 30 00:57 a/foobar

$ md5sum [34a]/foobar
d3b07a382ec010c01889250fce66fb13  3/foobar
d3b07a382ec010c01889250fce66fb13  4/foobar
d3b07a382ec010c01889250fce66fb13  a/foobar

Now we have 2 backups of a/foobar that are identical in all ways, including timestamp, but occupying different inodes.

One might think a solution would be --delete-before, which kills the benefit of the incremental scan but this doesn't help either as the file will not be deleted, but used as a basis in case incremental copy is possible.

One might further surmise then we can turn off this incremental-copy hedge with --whole-file, but this does not help the algorithm any, there's no way to get what we want.

I consider this behaviour another bug in rsync, where a beneficial behaviour could be construed from careful selections of various command arguments, but the desired outcome is not available.

A solution would unfortunately be moving from a single rsync as an atomic operation to a dry-run with -n, logging it, processing that log as input to manually pre-delete all changed files, then running rsync --link-dest to get what we want -- a big kludge compared to a single clean rsync.

Addendum: tried to pre-link $yesterday and $today on the backup server before the backup against production boxes with rsync --link-dest=../$yesterday $yesterday/ $today -- but the same result - any file that exists in any way, even 0 length, will never be removed and link-dested, instead a whole new copy will be made from the sourcedir with a new inode and using up more diskspace.

Looking at pax(1) as a possible pre-linking-before-backup solution.

  • I use --delete-after in this usage scenario, what's wrong with this? – gogoud Mar 30 '15 at 7:52
  • 1
    --delete-after is fine, but unrelated to the issue at hand. Files missing from the source will be deleted after the copy is done. The issue I'm elucidating relates to a backup being done today that is identical to yesterday's but against an old existing out of date file that is not linked to yesterday's inode, but stored as a new file at twice the total disk-space when yesterday's identical copy is considered. – math Mar 30 '15 at 14:10
  • Not entirely sure what you question is. Have you considered rsnapshot? Also, consider writing a small script to relink "identical" files. I do both on my systems. – roaima Apr 3 '15 at 8:29
  • 1
    If you don't get the answer you need here, you can post on the rsync list. The rsync developers regularly answer questions there along with many advanced users. You can find them via lists.samba.org/mailman/listinfo/rsync . I mostly lurk there and learn a lot. – Joe Apr 3 '15 at 22:32
  • rsnapshot wont recycle old backups - and I need to: if I have 2 mo and 2-mo+1 day old backups, I can cycle one in as the new target. Since ~5% of files change/day, I create 50K hardlinks instead of 10M. This speed diff allows backing up 5 servers/night vs not. hardlink(1) is slow (15x slower than rsync's metadata scan); pax is faster but thrashes HDD heads comparing old backup to new. rsync -n to get the delta list means hitting production servers twice (scanning 10M files is far more impact than copying the 50K changes). Ill mail the list about an option in rsync to allow this. – math Apr 6 '15 at 1:56
12

(Converted from question edit)

This is solved by upgrading rsync. Version 3.1.1 or later will now replace identical files in the target and --link-dest directory with one hardlinked file. Saves lots of space.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.