14

I'm using dd to copy data from an old hard drive to a new one. I want to be sure that the data's integrity is secure.

On this answer, Gilles says

If [dd] terminated successfully, then the backup is correct, barring a hardware fault…

What does that mean exactly? Does dd have some kind of built in verification?

If I were to use rsync instead, I run a second pass with --checksum as well, to verify. Is that kind of paranoia justified?

  • Define "integrity is secure". – Thorbjørn Ravn Andersen May 21 '17 at 12:22
  • @ThorbjørnRavnAndersen I mean that the copy is identical to the original. – Sparhawk May 21 '17 at 12:25
  • If you only have flat files, the traditional way for copying files is using tar or cpio. GNU tar has a verify flag: gnu.org/software/tar/manual/html_section/tar_81.html. These days rsync would probably be the simplest. – Thorbjørn Ravn Andersen May 21 '17 at 14:00
  • 1
    "barring a hardware fault" indicates that it doesn't do any verification. If it did, it could detect the hardware failure. – Barmar May 24 '17 at 18:57
18

dd or any other application does not have “some kind of built in verification” in the sense you're probably thinking of: it doesn't read back the data from the storage medium to compare with what was written. That's the job of the operating system.

It is not really possible to do a read-verify down to the hardware from an application. It would work in some scenarios, but in most cases it would achieve nothing. The application could read back what it just wrote if it's writing directly to a storage medium, but that would typically read back from an in-memory cache, which wouldn't give any useful assurance. In the example you cite, dd is writing to a pipe, and in that case it has no control over what happens to the data further down the line. In your rsync example, a second pass of rsync --checksum is pointless: in theory it could catch an error, but in practice, if an error does happen, then the second pass would probably not report anything wrong, so you're wasting effort on something that doesn't actually give a useful assurance.

However, applications do verify what happens to the data, in the sense that they verify that the operating system has accepted responsibility for the data. All system calls return an error status. If a system call returns an error status, the application should propagate that error to the user, generally by displaying an error message and returning a nonzero exit status.

Beware that dd is an exception: depending on the command line parameters, dd might ignore some errors. This is extremely unusual: dd is the only common command with this property. Use cat instead of dd, that way you don't risk corruption and it may well be faster.

In a chain of data copying, two kinds of errors can arise.

  • Corruption: a bit is flipped during the transfer. There is no way to verify this at the application level, because if that happens, it's due to a programming bug or hardware error that is highly likely to cause the same corruption when reading back. The only useful way to verify that no such corruption happened is to physically disconnect the media and try again, preferably on a different computer in case the problem was with the RAM.
  • Truncation: all the data that was copied was copied correctly, but some of the data was not copied at all. This one is worth checking sometimes, depending on the complexity of the command. You don't need to read the data to do that: just check the size.
  • I believe most storage mediums use enough FEC to detect + correct a single bit flip. – gardenhead May 20 '17 at 18:29
  • 2
    Of course if you copy an entire hard disk with dd and then immediately compare the hard disk you know it worked because the cache isn't big enough. – Joshua May 20 '17 at 22:25
  • 1
    Thanks for the answer (+1). I probably should mention I'm using a fairly basic dd if=/dev/sdc of=/dev/sdb bs=4M, so my understanding is that the issues of ignoring errors and speed (more or less, compared to cat) are moot. Are you saying to just check the size by mounting then df? – Sparhawk May 21 '17 at 0:51
4

No, dd does not do an explicit verification. If you want/need a forensically verified copy of your disk or any part of it, use dcfldd which is an enhanced version of dd developed by the U.S. Department of Defense Computer Forensics Lab.

4

The only way to be "sure" is to do an additional read-and-compare pass (after dropping caches).

Aside from that, dd detects read and write errors the same way all other programs do... it works if the drives (and other components involved) report errors; for drives that accept data silently w/o actually writing them you're out of luck.

Is that kind of paranoia justified?

If you can't trust your hardware to be reliable, things get complicated...

  • It's more complicated than this, both about read-and-compare and about dd detecting errors. – Gilles May 20 '17 at 13:20
  • Well, if you're going that far, dd has serious data corruption issues but special cases such as these were not part of the question. – frostschutz May 20 '17 at 14:47
  • Those corruption issues could justify verifying the data produced using dd. The real solution is to use anything but dd since silent data corruption is a specialty of dd. – Gilles May 20 '17 at 16:37
  • 2
    @Gilles, or just don't tell dd to ignore errors. You can't exactly blame a program for doing exactly what you asked it to. – Mark May 20 '17 at 20:24
  • @Mark And how, pray, to you tell dd not to ignore errors? And no, conv=noerror is not a correct answer. See frostschutz's answer for an example. I do blame the design of dd for making ignoring errors a default mode, and one that cannot be turned off without knowing its internal mechanics very precisely. – Gilles May 21 '17 at 19:07
2

Yes, faulty hardware may insert random error bits in the data at some rate as one bit per number of megabytes, this is possible and takes place in practice sometimes.

Usually, I use md5 or sha1 hash to verify the data is intact, by re-reading both the source and the destination, e.g.:

dd if=/dev/sdb of=~/hd_backup
dd if=/dev/sdb | md5sum
dd if=~/hd_backup | md5sum

This assumes that the data is much larger than the filesystem cache, otherwise, you may need to re-start the system to verify the actual data on the medium and not the cache contents, or use another system for it.

  • It is sufficient to unmount/mount the filesystem to force the OS to write the filesystem cache to the device. – miracle173 May 21 '17 at 16:17
  • miracle173, but even after syncing, doesn't the OS keep in cache what it wrote ? so i'm not sure unmounting will clear all cache from RAM. – Matt Dec 18 '18 at 15:02
1

From man dd:

When finished, dd displays the number of complete and partial input and output blocks, truncated input records and odd-length byte-swapping blocks to the standard error output.

A partial input block is one where less than the input block size was read. A partial output block is one where less than the output block size was written. Partial output blocks to tape devices are considered fatal errors. Otherwise, the rest of the block will be written. Partial output blocks to character devices will produce a warning message.

dd verifies the input/ouput block sizes match every time it copies a block. If they don't, it handles the error with a warning or fatal error (overidden with noerror). That's why dd works virtually all the time.

Still, it doesn't replace manually verifiying the integrity of your disk. If the information is valuable to you, then yes, your paranoia is justified. Run a manual verification once dd is finished.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.