rsync --delete not removing all deleted files

Question

I am using rsync for backing up several systems, either to external hard drives or over the network to another machine. The idea is to maintain a replica of the original filesystem, with a few things like /dev, /proc, /mnt, and what not excluded, which is then snapshotted at the destination (i.e. with btrfs or zfs). To that end, I use a command along the lines of

rsync -avP --delete --exclude /dev/* --exclude /proc/* --exclude /mnt/* etc. etc. / backup@fileserver:/mnt/array/backup/machine/

Which works reasonably well. However, I just noticed that rsync is not reliably deleting things on the destination. As a result, there are many more files on the destination than exist at any time on the source, in one case the back up was 100 GB larger than the source. The leftover files seem to be mostly old versions of various software and libraries. For example, on the original machine, /usr/include/qt/QtWidgets/ contains

5.9.2
QAbstractButton
qabstractbutton.h
....

But on the backup target, the same folder contains:

5.6.0
5.6.1
5.7.0
5.7.1
5.8.0
5.9.0
5.9.1
5.9.2
QAbstractButton
qabstractbutton.h

rsync seems to be copying over the new files just fine, but forgetting to remove all the old ones. The odd part is that some files are in fact removed (rsync always reports deleting a lot of files when synchronizing) but it seems that many files are not actually deleted. As a result, my backups are taking up a lot more space than they should, and they can't be directly restored because they can take up more space than the original partition to be due to all the extra files, and things like flat-file package databases are all screwed up, containing many duplicate entries for old versions packages.

Any ideas on what might be going on here? Is there some other flag I need to pass to rsync so that it actually deletes everything it's supposed to?

Answer 1

Some suggestions for things to try come from the contents of the man page (the section about --delete). There are a lot of caveats and warnings. Have you considered all of these?

• Works only "for the directories that are being synchronized". I guess this means they must have a parent/ancestor directory being copied.

• Similarly, it doesn't apply to anything being explicitly excluded:

Files that are excluded from the transfer are also excluded from being deleted unless you use the --delete-excluded option or mark the rules as only matching on the sending side.

• If the sending side detects any I/O errors, then the deletion of any files at the destination will be automatically disabled. Override with --ignore-errors.

• Try with --dry-run first? Combine with --verbose and closely examine everything that's happening with problem directories.

  This option can be dangerous if used incorrectly! It is a very good idea to first try a run using the --dry-run option (-n) to see what files are going to be deleted.

So consider trying --delete-excluded, --ignore-errors, and/or --dry-run/--verbose...not necessarily as a "fix" but at the very least there may be helpful data to be gleaned.

Also, take a look at the --delete-before flag. This is supposed to do the receive-side deletes before the actual transfer. Again, this might not fix everything but could be revealing. It'd be interesting to see if it helps reduce disk space usage, too.