6

I am using rsync for backing up several systems, either to external hard drives or over the network to another machine. The idea is to maintain a replica of the original filesystem, with a few things like /dev, /proc, /mnt, and what not excluded, which is then snapshotted at the destination (i.e. with btrfs or zfs). To that end, I use a command along the lines of

rsync -avP --delete --exclude /dev/* --exclude /proc/* --exclude /mnt/* etc. etc. / backup@fileserver:/mnt/array/backup/machine/

Which works reasonably well. However, I just noticed that rsync is not reliably deleting things on the destination. As a result, there are many more files on the destination than exist at any time on the source, in one case the back up was 100 GB larger than the source. The leftover files seem to be mostly old versions of various software and libraries. For example, on the original machine, /usr/include/qt/QtWidgets/ contains

5.9.2
QAbstractButton
qabstractbutton.h
....

But on the backup target, the same folder contains:

5.6.0
5.6.1
5.7.0
5.7.1
5.8.0
5.9.0
5.9.1
5.9.2
QAbstractButton
qabstractbutton.h

rsync seems to be copying over the new files just fine, but forgetting to remove all the old ones. The odd part is that some files are in fact removed (rsync always reports deleting a lot of files when synchronizing) but it seems that many files are not actually deleted. As a result, my backups are taking up a lot more space than they should, and they can't be directly restored because they can take up more space than the original partition to be due to all the extra files, and things like flat-file package databases are all screwed up, containing many duplicate entries for old versions packages.

Any ideas on what might be going on here? Is there some other flag I need to pass to rsync so that it actually deletes everything it's supposed to?

6

Some suggestions for things to try come from the contents of the man page (the section about --delete). There are a lot of caveats and warnings. Have you considered all of these?

  • Works only "for the directories that are being synchronized". I guess this means they must have a parent/ancestor directory being copied.

  • Similarly, it doesn't apply to anything being explicitly excluded:

Files that are excluded from the transfer are also excluded from being deleted unless you use the --delete-excluded option or mark the rules as only matching on the sending side.

  • If the sending side detects any I/O errors, then the deletion of any files at the destination will be automatically disabled. Override with --ignore-errors.

  • Try with --dry-run first? Combine with --verbose and closely examine everything that's happening with problem directories.

    This option can be dangerous if used incorrectly! It is a very good idea to first try a run using the --dry-run option (-n) to see what files are going to be deleted.

So consider trying --delete-excluded, --ignore-errors, and/or --dry-run/--verbose...not necessarily as a "fix" but at the very least there may be helpful data to be gleaned.

Also, take a look at the --delete-before flag. This is supposed to do the receive-side deletes before the actual transfer. Again, this might not fix everything but could be revealing. It'd be interesting to see if it helps reduce disk space usage, too.

| improve this answer | |
  • Hmm, maybe --ignore-errors is the ticket. Invariably, rsync runs in to some sort of error with a file disappearing before it can be copied or a gvfs mount point that can't be read by root. I'll try adding that flag and see what happens. – alex.forencich Oct 28 '17 at 19:49
  • 1
    Yep, --ignore-errors was the ticket. Thanks! – alex.forencich Oct 28 '17 at 19:54
  • I found, as @Gringo Suave said above as a comment, that wildcards were my issue. I had to use --dry-run and figure out how to rid the wildcard before --delete could work. – bafromca May 15 at 13:54
  • @bafromca What was the particular issue you faced? I think "wildcards can be problematic with rsync" is a bit misleading as it implies there's something tricky about rsync's handling of wildcards. The thing is, rsync doesn't handle them. That's what the shell is for. So make sure each wildcard is visible to the shell (i.e. don't single quote or escape any of them). Then it should be easy to reason about what will happen in rsync, e.g. --delete will only delete contents of directories you explicitly name so foo or foo/ but not foo/*. – B Layer May 16 at 0:28
  • 1
    @bafromca Cool. You've leveled up your "rsync fu", then. :) – B Layer May 17 at 17:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.