1

If I use rsync for an automated backup, how can I avoid it syncing files that have been corrupted at the source?

For example:

I have /source/*.* and it's periodically automatically synced to /destination/*.* where /source and /destination are different physical disks. Each time the files are updated, when rsync runs it copies the updated files to the destination.

How can I avoid it copying a file which has become corrupted due to HDD failure? Obviously I do not want the corrupted file copied to the backup drive.

There are two different modes for deciding what to sync.

the -c checksum swich which does a file checksum.

-c, --checksum
      This changes the way rsync checks if the files have been changed
      and  are in need of a transfer.  Without this option, rsync uses
      a "quick check" that (by default) checks if each file’s size and
      time of last modification match between the sender and receiver.
      This option changes this to compare a 128-bit checksum for  each
      file  that  has a matching size.  Generating the checksums means
      that both sides will expend a lot of disk I/O  reading  all  the
      data  in  the  files  in  the transfer (and this is prior to any
      reading that will be done to transfer changed  files),  so  this
      can slow things down significantly.

and the -u switch flag:

-u, --update
  This  forces  rsync  to  skip  any  files  which  exist  on  the
  destination  and  have  a  modified  time that is newer than the
  source  file.   (If  an  existing   destination   file   has   a
  modification time equal to the source file’s, it will be updated
  if the sizes are different.)

Clearly -c will not have the desired behaviour in this case. If the source file is corrupted it will have a different checksum and be copied to the backup.

My assumption is that -u will do exactly what I want. Because the corrupted file will have the same mtime as the version on the backup drive, if the source is corrupted it will not be copied across.

My question is, is using -u here to copy files an adequate backup strategy?

Alternatively, should I consider a different approach or a different tool entirely?

  • 2
    Do you only ever store a single backup of your files (the most recent one)? You could instead have incremental updates using --link-dest into a sequence of backup directories. That way you would back up your corrupted files, but still have an older backup that wasn't corrupted. Likewise if you use a proper backup solution like restic or borgbackup which by default saves a separate "snapshot" per backup. Not an answer since I'm not showing how to actually do things. – Kusalananda Nov 30 '19 at 14:09
  • To add a bit more background, most of the files are videos and photos which are very unlikely to change over time (I might change the metatdata on the photos occasionally to make them easier to manage) but they're not something I'll check frequently. A snapshot solution might be worse depending on setup as keeping a year's worth of snapshots, if a file is corrupted at source and I don't check it on the source for a year, then the backup contains only corrupted files unless I store a complete history Admittedly it's unlikely that a faulty source HDD wouldn't become obvious within a year. – Tom B Nov 30 '19 at 14:17
  • 1
    The --link-dest option to rsync could be used to hard link files that have not changed, rather than copying the again (indeed, this is the point of this option). Both restic and borgbackup would additionally do deduplication of data. I'm using restic, and backing up five computers at home for over a year in a single backup repository uses much less space than say, Time Machine on macOS (which is using a similar approach to rsync --link-dest), doing the same job. – Kusalananda Nov 30 '19 at 14:22
1

There is no way to identify files that have been "corrupted at source". What you may be able to do is to identify files that have been written to one or more disk blocks that can no longer be read, but this will only be a subset of the possible set matching your criterion.

As you have suggested, the -u (--update) flag will instruct rsync to avoid attempting to read and update any file either that has a newer modification time on the destination regardless of any other attribute, or that has the same modification time and size.

What might be better is to use rsnapshot for making your backup copies to an ext4 target (not NTFS or FAT/exFAT), and to use smartctl to implement SMART disk checking on your source.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.