3

I am using scp to transfer some large files (> 1 GB) from my laptop (Ubuntu 12.04) to my desktop (Ubuntu 14.04). The laptop has a wireless connection to my router and the desktop has a cable connection.

Out of curiosity I decided to verify the transferred files using an md5 digest: I created the digest on the source computer, transferred it to the target computer, and used it with the -c option to verify the transferred files. To my surprise, more than 50% of the times I get an error:

md5sum: WARNING: 1 computed checksum did NOT match

My question: does data corruption occur so often with scp? I would expect that (1) data transfer errors are rather rare, and (2) scp checks its data and possibly resends it in case of error. What can be the cause of these errors? My hypotheses:

  1. Bad WLAN connection that frequently corrupts data during transfer.
  2. Bad memory: md5sum -c fails even thought the data is correct because the RAM is not working properly.
  3. Bad hard disk: md5sum -c fails because the disk contains error.
  4. A combination of the above reasons.

Note that my hard disk and RAM are pretty new: I bought them 5 months ago.

1
  • Problem solved: following the indications in the accepted answer I check my desktop's memory and found out it was not working properly. I replaced the RAM and then transferred the files again: all md5 files verify correctly now.
    – Giorgio
    Jul 21 '14 at 6:56
5

scp transfers files over SSH, which does cryptographic authenticity & integrity checking. That basically rules out the bad WLAN possibility.

Bad memory is reasonably likely. Note that bad memory often starts bad, it's not typically from getting old. Installing and running memtest86/metest86+ will either confirm this or mostly rule it out. (For ruling it out, you want to leave the test running for a while, at least overnight). If it finds an error, you don't need to keep it running, you can stop immediately and proceed to replacing DIMMs.

The disk corrupting it is also possible. Similarly, you could have bad cabling to the disk, or a defective controller, etc.

Other possibilities are filesystem bugs (unlikely if you're using something common like ext4), malware (thankfully fairly uncommon on Linux), but this is most likely a hardware problem.

10
  • Thanks a lot for the hints. The partition on which I have copied the files is formatted with a ext4 file system. I am transferring the files to the same computer to another mounted disk, to see if the problem occurs again. After that I will run a memory test.
    – Giorgio
    Jun 16 '14 at 18:35
  • Actually, I have just run another test, and one file that did not check now seems to be correct. Summarizing: first check NOT OK, several subsequent checks OK. I did not change the file nor the md5 file, just ran the check several times. So this should point to bad memory.
    – Giorgio
    Jun 16 '14 at 18:39
  • @Giorgio It could also be the disk reading it back wrong, or a defective CPU, but I'd go with bad memory as most likely. Note that if you do have bad memory, it has been silently corrupting your data for a while...
    – derobert
    Jun 16 '14 at 18:41
  • I know. This has been going on for a few months. Luckily most of my important data is source files. I also have an old backup on disk which was done on the old computer. I might check it against my current data after I have solved this problem (provided memory turns out to be the problem).
    – Giorgio
    Jun 16 '14 at 18:44
  • 1
    Memory does get bad over time. I don't recall ever getting new bad memory (but I do know people who've had it happen). I have several times retired months- or years-old memory sticks because they'd gone bad. Jun 16 '14 at 23:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.