5

I rsynced a large file from remote centos to local ubuntu with

rsync -avzP user@<remote-ip>:/path/to/file .

It reported the transfer went well:

sent 30 bytes  received 257,293,476 bytes  1,296,188.95 bytes/sec
total size is 8,217,194,015  speedup is 31.94

As far as I know rsync automatically verifies the transfer went well with hash checks after the transfer is completed.

Out of curiosity I computed md5 hashes on centos and ubuntu, and these are different:

centos: 0faa300b7b0b81bfe65199da932eb6e2
ubuntu: f3a0fcc59516d4e68fd207bdbb1fc169

Both hashes are computed with md5sum:

centos> md5sum --version
md5sum (GNU coreutils) 8.22

ubuntu> md5sum --version
md5sum (GNU coreutils) 8.25

So the verisons are a little different, but can that lead to a different values of the hashes?

Edit:

Here are ls -l output:

centos: -rw-rw-r--.  1 username username 8217194015
ubuntu: -rw-rw-r--   1 username username 8217194015

Centos output includes mysterious dot I've never heard about. (could it be related to lvm? lvm is used on that centos)

Edit 2:

Checking md5sum -b leads to different results as well:

centos: 0faa300b7b0b81bfe65199da932eb6e2
ubuntu: 6d799f6981066d82c7f861576b4980e1

What hash algorithm does rsync use? According to wikipedia rsync uses md5 to check if the chunk is the same:

The recipient splits its copy of the file into chunks and computes two checksums for each chunk: the MD5 hash, and a weaker but easier to compute 'rolling checksum'. It sends these checksums to the sender. The sender quickly computes the rolling checksum for each chunk in its version of the file; if they differ, it must be sent. If they're the same, the sender uses the more computationally expensive MD5 hash to verify the chunks are the same.

  • What did you run md5sum on exactly? You copied a directory, but md5sum acts on file contents, not on directories. – Gilles Dec 18 '16 at 22:41
  • @Gilles: I copied a file and run md5sum on that file. Why do you say that I've copied a directory? – Adobe Dec 18 '16 at 22:45
  • The source of the copy is .; that's always a directory. Is the file you checked actually one of the ones you copied? Give us the exact commands (including file and directory names) and their exact output instead of truncating. The problem is highly likely to be because something in what you truncated matters even though you thought it didn't matter. – Gilles Dec 18 '16 at 22:49
  • @Gilles: when writing a question, I've mistaken the order -- it was rsync -avzP user@<remote-ip>:/path/to/file . -- I copied a file from remote machine to the local machine. But is was a file. This is not a mistake. – Adobe Dec 18 '16 at 22:58
1

There's a wrong assumption here:

As far as I know rsync automatically verifies the transfer went well with hash checks after the transfer is completed.

Rsync uses checksums to determine if a sync is needed. But, Rsync does not reread the created copy, it trust the kernel to report errors. So, the conclusion is simple: the files are not identical. Could be just one bit, could be more. How much mismatch there is, a checksum doesn't tell you.

0

The dot . means, that the file has SELinux context, as every file on CentOS (and no files on Ubuntu), which might confuse md5sum in some way. Did you try to run md5sum with b switch to make sure it will not get mangled by the "to-text" conversions?

  • With md5sum -b flag centos shows the same checksum as before, but ubuntu shows value different from text mode, and also different from centos value: ubuntu: 6d799f6981066d82c7f861576b4980e1. – Adobe Dec 18 '16 at 9:59
  • 1
    An SELinux context has nothing to do with md5sum -b. The md5sum utility only cares about file contents, not about metadata such as SELinux contests. The -b flag only matters on Windows, on Linux it's just ignored. – Gilles Dec 18 '16 at 22:43
  • @Gilles well, yes, you are obviously right about the md5sum, but the question was also about the . in the output of ls. There is certainly something fishy about the ubuntu system, since it prints different hash each time. – Jakuje Dec 22 '16 at 21:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.