3

I have a backup script which uses rsync -avz --link-dest=$oldbkp $source $newbkp at its core. The problem is that rsync a lot of times doesn't recognize that a file in $source hasn't changed and so it plainly copies it to $newbkp instead of hard linking it from $oldbkp. Another perplexing thing is that it is inconsistent - there are some files where the hardlinking works as expected. The backup partition is ntfs.

As you can see here, a backup only an hour after the previous one takes whole 2GB of new data when the content barely changed at all (this is my home PC).

$ du -hsc 20170424-1559 20170424-1724
2.6G    20170424-1559
2.1G    20170424-1724
4.6G    total

I've tried stating some examples. This one is failed hardlink (sha256 is the same for all):

$ stat 20170424-1559/Documents/depeche 
  File: 20170424-1559/Documents/depeche
  Size: 21400           Blocks: 48         IO Block: 4096   regular file
Device: 811h/2065d      Inode: 140380      Links: 1
Access: (0777/-rwxrwxrwx)  Uid: ( 1000/   marek)   Gid: ( 1000/   marek)
Access: 2017-04-24 17:14:00.271104500 +0200
Modify: 2016-08-01 16:30:38.000000000 +0200
Change: 2017-04-24 15:59:44.407252700 +0200
 Birth: -

$ stat 20170424-1724/Documents/depeche 
  File: 20170424-1724/Documents/depeche
  Size: 21400           Blocks: 48         IO Block: 4096   regular file
Device: 811h/2065d      Inode: 361117      Links: 1
Access: (0777/-rwxrwxrwx)  Uid: ( 1000/   marek)   Gid: ( 1000/   marek)
Access: 2017-04-24 17:24:55.732080500 +0200
Modify: 2016-08-01 16:30:38.000000000 +0200
Change: 2017-04-24 17:24:55.736274500 +0200
 Birth: -

$ stat ~/Documents/depeche
  File: /home/marek/Documents/depeche
  Size: 21400           Blocks: 48         IO Block: 4096   regular file
Device: 2ah/42d Inode: 4397        Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1000/   marek)   Gid: ( 1000/   marek)
Access: 2017-03-07 09:51:07.681090473 +0100
Modify: 2016-08-01 16:30:38.000000000 +0200
Change: 2016-11-06 19:58:14.053859011 +0100
 Birth: -

This one is successful hardlink (sha256 are the same):

$ stat 20170424-1559/Documents/ios7bkplist.txt 
  File: 20170424-1559/Documents/ios7bkplist.txt
  Size: 1983            Blocks: 8          IO Block: 4096   regular file
Device: 811h/2065d      Inode: 344437      Links: 4
Access: (0777/-rwxrwxrwx)  Uid: ( 1000/   marek)   Gid: ( 1000/   marek)
Access: 2017-04-24 15:59:44.574850700 +0200
Modify: 2016-04-04 22:03:55.000000000 +0200
Change: 2017-04-24 17:24:56.022250400 +0200
 Birth: -

$ stat 20170424-1724/Documents/ios7bkplist.txt 
  File: 20170424-1724/Documents/ios7bkplist.txt
  Size: 1983            Blocks: 8          IO Block: 4096   regular file
Device: 811h/2065d      Inode: 344437      Links: 4
Access: (0777/-rwxrwxrwx)  Uid: ( 1000/   marek)   Gid: ( 1000/   marek)
Access: 2017-04-24 15:59:44.574850700 +0200
Modify: 2016-04-04 22:03:55.000000000 +0200
Change: 2017-04-24 17:24:56.022250400 +0200
 Birth: -

$ stat ~/Documents/ios7bkplist.txt            
  File: /home/marek/Documents/ios7bkplist.txt
  Size: 1983            Blocks: 8          IO Block: 4096   regular file
Device: 2ah/42d Inode: 4413        Links: 1
Access: (0777/-rwxrwxrwx)  Uid: ( 1000/   marek)   Gid: ( 1000/   marek)
Access: 2017-02-28 20:03:32.858085513 +0100
Modify: 2016-04-04 22:03:55.000000000 +0200
Change: 2016-11-06 19:58:14.550522987 +0100
 Birth: -

Basically the same thing happens when I use -c with rsync to force long file checking.

Is there anything I'm overlooking?

3

The problem is the following (from man rsync):

... The files must be identical in all preserved attributes (e.g. permissions, possibly ownership) in order for the files to be linked together.

In your case, the permissions of the files are (from your examples)

Access: (0644/-rw-r--r--) # hardlink failed (original)
Access: (0777/-rwxrwxrwx) # hardlink failed (backup)

Access: (0777/-rwxrwxrwx) # hardlink created (original)
Access: (0777/-rwxrwxrwx) # hardlink created (backup)

For instance

$ chmod 777 A/file
$ rsync -az A/ B/
$ chmod 644 A/file
$ rsync -az --link-dest=$PWD/B/ A/ C/

results in

$ du -hsc A B C
965M    A
965M    B
965M    C
2.9G    total

while, resetting permissions to original, results in

$ chmod 777 A/file
$ rsync -az --link-dest=$PWD/B/ A/ D/
$ du -hsc A B D
965M    A
965M    B
4.0K    D
1.9G    total

You can try with some file to get it to work (I guess with --size-only, which should skip files of equal size).

What you should do is try to figure out if you changed permissions after the last backup, or, if not, why permissions changed in your backup directory.

  • Oh, I completely missed those permissions. You're correct. There is another problem though (and this is also a reason why I leniently didn't bother to check them) - it seems that rsync for some reason changes them. If you look at the depeche file and its perms, they are 644 at home, but copied over to the backup, rsync changed them to 777 without my permission. (I didn't do that between those backups) So essentialy he's blinding himself. I didn't thought it would do that. I thought ntfs supported posix permissions? – JonnyRobbie Apr 25 '17 at 1:02
  • Not fully, I suppose. Well, I guess --no-perms should be a suitable workaround. Thanks. – JonnyRobbie Apr 25 '17 at 1:11
  • I don't know if it helps, but take a look at this answer. – resc Apr 25 '17 at 8:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.