Is it safe to use rsync when the source filesystem is mounted and writeable, assuming that it may very well be modified while rsync is running? I presume that under race conditions I may not get the latest modifications, but is there any risk of corruption?

  • 1
    Are you concerned about consistency (between files, such as a data file and some associated metadata file, log or some such) or corruption (within a single file) in the event of changes? – user Sep 11 '13 at 13:43
  • Good point, I hadn't considered consistency between files. I don't suppose rsync can really do anything about that, only a snapshot can, but I'm more worried about corruption within a file. – brianmearns Sep 11 '13 at 14:04

Using rsync is fairly safe on read-write mounted file systems. When rsync is started it builds up a file list and then starts to copy those files. This file list is not being updated during the run. The actual data is then copied.

This means when a file changes after rsync has built the file list, it will copy the new content. However, when a new file is added after rsync has built up its file list, this new file is not copied. If a file is deleted after rsync has built the list of files, rsync will warn that it could not copy that file. Keep in mind that rsync is not a snapshot, this means it's hard to tell at which point in time rsync copied the data.

File system or volume manager snapshots (e.g. using ZFS or LVM), on the other hand, are created instantly and are a consistent snapshot of the file system at a well defined point in time which can then be copied to another host. rsync does not offer this kind of consistency.

EDIT: As others have pointed out in the comments, there's a chance that rsync might actually corrupt your file. When rsync starts to read a file and an application is writing that file at the same time, you might end up with a corruped file.

| improve this answer | |
  • 2
    Added a note on deleted files. Would probably give this another upvote if I could for the volume manager snapshot + rsync suggestion. The only thing I feel this answer is missing is the behavior of rsync when files on the source are changed in the middle of being copied. I doubt it handles that well, though. – user Sep 11 '13 at 15:18
  • Agreed. If you want to be certain files are consistent, you need operating system support? – Thorbjørn Ravn Andersen Sep 11 '13 at 20:03
  • 1
    This answer applies provided that the application on the source side first writes to a temporary file, then moves it into place. If the application opens an existing file for writing while rsync happens to be reading that file, rsync may copy a corrupted mix of the old and the new versions. – Gilles 'SO- stop being evil' Sep 11 '13 at 23:09
  • @Gilles Thanks for pointing that out. I updated my answer. Is this related to rsync or does this occur to any application which reads while another application writes to the same file? – Marco Sep 11 '13 at 23:22
  • 1
    @Marco That would happen with any application, unless both use some locking protocol (unix doesn't have mandatory locks, only advisory locks, i.e. locks that only work if everybody is using them). That's why the normal protocol for modifying a file if concurrent modification is a concern is to write a temporary file then move it into place. – Gilles 'SO- stop being evil' Sep 11 '13 at 23:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.