I use a simple script to create incremental backups with rsync from my local linux station (XFS file system) to an external HD (NTFS file system) attached to it:

TIMESTAMP=`date "+%Y-%m-%dT%H-%M-%S"`

# Create new backup using rsync and output to log
rsync -avPh --delete --link-dest=$TARGETDIR/$USER-Current $SOURCEDIR/$USER/ $TARGETDIR/$USER-$TIMESTAMP > /archive/Backup-Logs/$USER-$TIMESTAMP.log 2>&1
# check exit status to see if backup failed
    if [ "$?" = 0 ]; then
# Remove link to current backup
        rm -f $TARGETDIR/$USER-Current
# Create link to the newest backup
# Rename directory if failed

I checked the resulting backup directories created in this way using 'du -h -d1'. I find that the first backup directory created is indeed the biggest and the ones thereafter are smaller, however, the subsequent directories are all about the same size. That is to say, I never find a latter directory to be substantially smaller than than a previous one, even if 'essentially' no files have changed between them. Based on the sizes I am seeing the backups seem more differential (versus the very first backup) than incremental.

Even worse, for my other set of backups, each of the backup directories is the same as the initial backup. What am I missing?

  • Some actual sizes would be good to see. You may want to add -H to preserve hard links, and -S to preserve sparseness of sparse files. Also, directories do take space depending on how many files are in them, so even just creating the directory structure and hard-linking all files to the previous backup can never create a backup of zero size. Additionally, double quote all variable expansions!! And finally: run df on all backup directories at once, not an individual directory. df takes hard links into account. Not an answer as I don't see the actual numbers you allude to. – Kusalananda May 3 at 7:53
  • Is there really a space in the filename /archive/Backup Logs/...? If so you have an extra dest arg to your rsync. Use ls -l on a file you think hasn't changed in a backup, and the 2nd column is the number of hard links. It shouldn't be 1 if it got linked to a previous backup. – meuh May 3 at 8:28
  • Thanks to you both for the suggestions. OK, so I can confirm the links are not all getting made -- indeed most are '1'. Now -- here's the key -- if I run this same script (no, there's not a space in /archive/Backup Logs/ -> /archive/Backup-Logs/) but point to a different target drive, it works. The new drive is XFS, whereas the old drive is NTFS. I did try to run the script to the NTFS using '--modify-window=1' but still no luck. What are your thoughts? – dtx1 May 3 at 21:33

I would say that your solution is probably doing what it should.

There are some code concerns - you should have all your variables inside double-quotes, for example ln -s "$TARGETDIR/$USER-$TIMESTAMP" "$TARGETDIR/$USER-Current". And as far as I can see you've reinvented rsnapshot.

I think the confusion comes from the file hard links. A hard link will make it not only seem like a file is in a directory, but to all intents and purposes it is in that directory. Each of your backups will seem like it's a copy of the entire backup set, but when you look more closely, files that are unchanged between backup sets use only one instance of storage. If you have a 100GB backup and 90% remain unchanged, you could end up with two backup sets taking only 110GB between them - but each one uses 100GB when measured separately.

The du command will only count file usage once, so you can use that across your entire backup hierarchy to see what's actually different between backup sets.


# The same file, linked
ls -l top/dir?/*
-rwxrwx---+ 2 roaima roaima 1690442768 May  2 17:56 top/dir1/file.mp4
-rwxrwx---+ 2 roaima roaima 1690442768 May  2 17:56 top/dir2/file.mp4

# First directory
du -hs top/dir1
1.6G    top/dir1

# Second directory
du -hs top/dir2
1.6G    top/dir2

# Both directories, but each file is counted only once
du -hs top/dir?
1.6G    top/dir1
0       top/dir2

# All directories, but each file is counted only once
du -hs top
1.6G    top

In your instance try this

du -hs "$TARGETDIR"/*
| improve this answer | |
  • thanks for pointing towards rsnapshot and your other advice – dtx1 May 5 at 2:43

Before doing the rsync, for an incremental, I do cp -al which give a reference point for the rsync. cp -al (archive with hard link instead of copying). The original file will stay as long as you have one or more hard links pointing to it.

eg. cp -al lastbkup newbackup then rsync against the newbackup

| improve this answer | |

There are problems rsyncing to an NTFS-3g, as by default the file permissions are usually all converted to 777 (rwx to all). So perhaps when you copy file A with, say, permissions 644, the first copy becomes 777 A. Later copies then fail to link to this, as the original file is still 644 A, even though the second copy is 777 A. rsync must be comparing the source permissions with the link-dest and find the mismatch.

One option is to replace -a which include preserving user, group, and permissions, with -rlt.

Alternatively, mount the filesystem with extra option permissions, (or use a window-linux user map file) which preserves Posix permissions.

| improve this answer | |

thanks to everyone for the help. I decided to format the external drive as XFS, which is the same as the source drive. And it worked! I can confirm the size of the first and subsequent backups (via du -h -d1) as:

117G    ./Apps-2020-05-04T14-19-42
3.5G    ./Apps-2020-05-04T15-07-12
121G    .
| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.