4

This question already has an answer here:

I need to transfer files from several servers to a single remote host using Rsync over SSH. However, I need to be sure that before the file is deleted on the source using the --remove-from-source argument, that the file transferred is actually there.

From what I've been reading, there are no post transfer checksums and rsync trusts the kernel response but those articles were dated back from 2005-2009. I was wondering if this has changed in recent updates to rsync? If not, what method(s) are there to check this and then delete the source file once verified?

Edit: I don't get how this is a duplicate. My question does not pertain to local drives on the same system...

marked as duplicate by Drav Sloan, cuonglm, Marco, chaos, Anthon Aug 11 '15 at 17:07

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • I doubt rsync has changed substantially since then but if the packet was mangled en route I doubt SSH would be able to get an intelligible result after trying to decrypt it, not to mention the CRC in the TCP packet. – Bratchley Aug 11 '15 at 16:08
  • 1
    rsync over ssh should be immune to network data corruption. If it's not, that's a security hole in SSH. – Peter Cordes Aug 11 '15 at 17:10
  • Aaaaa why does stackexchange only tell me this got closed after I spend an hour on a comprehensive answer!? – Anko Aug 11 '15 at 18:10
  • Very frustrating.. – linguru772x Aug 11 '15 at 18:18
  • I agree that sometimes we close too much and too fast. I nominated this question to reopen (although to be perfectly honest I don't believe it will happen). – jimmij Aug 11 '15 at 18:50
1

Summary: If rsync gets data to a disk, it will do so losslessly. However, to be totally sure it actually got data to the disk, you'll need to apply the fsync.diff patch, or call sync <files> afterwards.


SSH provides data integrity—you're receiving the same data as you're sending. That accounts for the network.

Then, rsync uses the write system call, asking the kernel to write the data to disk. Unless your hard disk is failing (a different question), this also preserves data integrity.

However, ensuring that data is actually on disk now is annoyingly not quite that simple. The write manual page makes the following note:

A successful return from write() does not make any guarantee that data has been committed to disk. In fact, on some buggy implementations, it does not even guarantee that space has successfully been reserved for the data. The only way to be sure is to call fsync(2) after you are done writing all your data.

I downloaded the most recent (3.1.2pre1) rsync's source code, grepped for fsync and got nothing. By default, rsync does not call fsync (I also grepped for the metadata-less version fdatasync: also nothing). This means whether those writes have done anything yet is filesystem-dependent.

As a solution, you can either:

  • Run sync <files>, which calls fsync on the given files. When that returns, they're definitely on disk.

  • Download the rsync source patches directory (provided as a separate download). Apply the fsync.diff patch by Sami Farin. It “lets you specify --fsync if you want fsync() to be called on every file we write”. (This will hopefully become default in the future.)

Usually though, modern filesystems do your writes Pretty Soon™, only briefly taking advantage of their freedom to cache when IO load is high. If you know your system, you might be OK with skipping this step. But do keep in mind when writing code for wider use that results may differ based on your filesystem, how it's tuned, and whether the on-drive firmware god is feeling benevolent.

Not the answer you're looking for? Browse other questions tagged or ask your own question.