rsync option to exclude partial files

Question

We are using rsync to sync two folders on same machine.

Files will be written to a source folder from another application. We have the problem that, even if a file is not completely written/copied to the source folder, rsync copies that file to destination.

Is there any way/option to check/transfer only complete files from the source folder

Answer 1

It's impossible to exclude partial files because there's no such concept. As soon as the producer has created the file, the file exists, but it starts out empty and gets filled gradually.

You can test whether the file is open for writing; that would tell you that it's incomplete. However this is not reliable: if the producer crashes (either the process crashes or the whole system crashes), you're left with an incomplete file that looks complete.

What you should do is define a protocol for the producer to mark the file as complete. The normal way to do this (and pretty much the only sane way) is for the producer to create the file in a temporary location, then move it into place (with a rename system call or the mv shell command) when it's finished.

You can use a naming convention:

• Producer: write to $FILENAME.tmp, then move to the final file.

  generate_data >"dir/$FILENAME.tmp"
  mv "dir/$FILENAME.tmp" "dir/$FILENAME"
  

• Consumer: exclude .tmp files.

  rsync -a --exclude='*.tmp' dir remote:
  

Or you can use a staging directory.

• Producer: write to the staging directory, then move to the final location.

  generate_data >"dir/staging/$FILENAME"
  mv "dir/staging/$FILENAME" "dir/"
  

• Consumer: exclude the staging directory.

  rsync -a --exclude='/staging' dir remote:
  

Answer 2

If the sizes of the files are fixed (after the write operation of the application), you can transfer only the files based on the size so the files that are not done being written to yet will not be copied :

--max-size=SIZE         don't transfer any file larger than SIZE
--min-size=SIZE         don't transfer any file smaller than SIZE

options of rsync provides that.

Alternately you can use fuser or lsof to check if the application is writing to the file at the instant of start transferring :

if fuser /path/to/file.txt >/dev/null 2>&1; do
    rsync ....
else
    sleep 10
fi

Answer 3

I use the following command which finds all files not modified in the last 30 minutes.

ssh sourceServer "find sourceDir -mmin +30 -type f | xargs -i -r rsync [ --remove-sent-files ] -aP {} destServer:destPath/"