22

I'm using dd to copy data from an old hard drive to a new one. I want to be sure that the data's integrity is secure.

On this answer, Gilles says

If [dd] terminated successfully, then the backup is correct, barring a hardware fault…

What does that mean exactly? Does dd have some kind of built in verification?

If I were to use rsync instead, I run a second pass with --checksum as well, to verify. Is that kind of paranoia justified?

4
  • Define "integrity is secure". Commented May 21, 2017 at 12:22
  • @ThorbjørnRavnAndersen I mean that the copy is identical to the original.
    – Sparhawk
    Commented May 21, 2017 at 12:25
  • If you only have flat files, the traditional way for copying files is using tar or cpio. GNU tar has a verify flag: gnu.org/software/tar/manual/html_section/tar_81.html. These days rsync would probably be the simplest. Commented May 21, 2017 at 14:00
  • 1
    "barring a hardware fault" indicates that it doesn't do any verification. If it did, it could detect the hardware failure.
    – Barmar
    Commented May 24, 2017 at 18:57

5 Answers 5

22

dd or any other application does not have “some kind of built in verification” in the sense you're probably thinking of: it doesn't read back the data from the storage medium to compare with what was written. That's the job of the operating system.

It is not really possible to do a read-verify down to the hardware from an application. It would work in some scenarios, but in most cases it would achieve nothing. The application could read back what it just wrote if it's writing directly to a storage medium, but that would typically read back from an in-memory cache, which wouldn't give any useful assurance. In the example you cite, dd is writing to a pipe, and in that case it has no control over what happens to the data further down the line. In your rsync example, a second pass of rsync --checksum is pointless: in theory it could catch an error, but in practice, if an error does happen, then the second pass would probably not report anything wrong, so you're wasting effort on something that doesn't actually give a useful assurance.

However, applications do verify what happens to the data, in the sense that they verify that the operating system has accepted responsibility for the data. All system calls return an error status. If a system call returns an error status, the application should propagate that error to the user, generally by displaying an error message and returning a nonzero exit status.

Beware that dd is an exception: depending on the command line parameters, dd might ignore some errors. This is extremely unusual: dd is the only common command with this property. Use cat instead of dd, that way you don't risk corruption and it may well be faster.

In a chain of data copying, two kinds of errors can arise.

  • Corruption: a bit is flipped during the transfer. There is no way to verify this at the application level, because if that happens, it's due to a programming bug or hardware error that is highly likely to cause the same corruption when reading back. The only useful way to verify that no such corruption happened is to physically disconnect the media and try again, preferably on a different computer in case the problem was with the RAM.
  • Truncation: all the data that was copied was copied correctly, but some of the data was not copied at all. This one is worth checking sometimes, depending on the complexity of the command. You don't need to read the data to do that: just check the size.
3
  • I believe most storage mediums use enough FEC to detect + correct a single bit flip.
    – gardenhead
    Commented May 20, 2017 at 18:29
  • 2
    Of course if you copy an entire hard disk with dd and then immediately compare the hard disk you know it worked because the cache isn't big enough.
    – Joshua
    Commented May 20, 2017 at 22:25
  • 2
    Thanks for the answer (+1). I probably should mention I'm using a fairly basic dd if=/dev/sdc of=/dev/sdb bs=4M, so my understanding is that the issues of ignoring errors and speed (more or less, compared to cat) are moot. Are you saying to just check the size by mounting then df?
    – Sparhawk
    Commented May 21, 2017 at 0:51
8

No, dd does not do an explicit verification. If you want/need a forensically verified copy of your disk or any part of it, use dcfldd which is an enhanced version of dd developed by the U.S. Department of Defense Computer Forensics Lab.

3
  • Thank you for introducing me to this.
    – Drakes
    Commented Mar 5, 2021 at 22:39
  • That's interesting. But sadly it doesn't have flag to verify data after write. You need to generate checksums during copy, and then generate chacksums on second pass. Commented Aug 24, 2023 at 19:50
  • 1
    Update. This utility should not be used anymore, at state of 2023. It's not suitable fror SSDs and NVMe's specifically. Max speed I had with it was 190 MiB/s after tuning bs, and even lower before palying with buffer size. But with dd and buffer of 32M I was able to acheive 2GiB/sec. So basically dd + sha256sum twice gave me far faster result. Can also use github.com/Xfennec/progress to track progress of sha256sum Commented Sep 7, 2023 at 20:04
6

The only way to be "sure" is to do an additional read-and-compare pass (after dropping caches).

Aside from that, dd detects read and write errors the same way all other programs do... it works if the drives (and other components involved) report errors; for drives that accept data silently w/o actually writing them you're out of luck.

Is that kind of paranoia justified?

If you can't trust your hardware to be reliable, things get complicated...

9
  • It's more complicated than this, both about read-and-compare and about dd detecting errors. Commented May 20, 2017 at 13:20
  • Well, if you're going that far, dd has serious data corruption issues but special cases such as these were not part of the question. Commented May 20, 2017 at 14:47
  • 2
    @Gilles, or just don't tell dd to ignore errors. You can't exactly blame a program for doing exactly what you asked it to.
    – Mark
    Commented May 20, 2017 at 20:24
  • 1
    Proving Non-Existence [of corruption issues in dd or anywhere] is hard (and some would say, the wrong thing to ask for); usually if you claim that dd is unsafe, you're supposed to provide evidence. I agree dd has a plethora of confusing options, but that's not what the question was about. The question was "I want to be sure that the data's integrity is secure. Does dd have some kind of built in verification?" - in that regard it's the same difference if you use cat, cp, ddrescue, ... - they all don't, it's just read and write but not verify for all of them. Commented May 21, 2017 at 19:45
  • 1
    Verification in general is a new fad (such as checksumming in ZFS, btrfs) usually we just rely on both software and hardware to not mess things up for us (and whatever error checking capabilities they have built in) and it works surprisingly well most of the time. Commented May 21, 2017 at 19:48
6

Yes, faulty hardware may insert random error bits in the data at some rate as one bit per number of megabytes, this is possible and takes place in practice sometimes.

Usually, I use md5 or sha1 hash to verify the data is intact, by re-reading both the source and the destination, e.g.:

dd if=/dev/sdb of=~/hd_backup
dd if=/dev/sdb | md5sum
dd if=~/hd_backup | md5sum

This assumes that the data is much larger than the filesystem cache, otherwise, you may need to re-start the system to verify the actual data on the medium and not the cache contents, or use another system for it.

2
  • It is sufficient to unmount/mount the filesystem to force the OS to write the filesystem cache to the device.
    – miracle173
    Commented May 21, 2017 at 16:17
  • miracle173, but even after syncing, doesn't the OS keep in cache what it wrote ? so i'm not sure unmounting will clear all cache from RAM.
    – Matt
    Commented Dec 18, 2018 at 15:02
1

From man dd:

When finished, dd displays the number of complete and partial input and output blocks, truncated input records and odd-length byte-swapping blocks to the standard error output.

A partial input block is one where less than the input block size was read. A partial output block is one where less than the output block size was written. Partial output blocks to tape devices are considered fatal errors. Otherwise, the rest of the block will be written. Partial output blocks to character devices will produce a warning message.

dd verifies the input/ouput block sizes match every time it copies a block. If they don't, it handles the error with a warning or fatal error (overidden with noerror). That's why dd works virtually all the time.

Still, it doesn't replace manually verifiying the integrity of your disk. If the information is valuable to you, then yes, your paranoia is justified. Run a manual verification once dd is finished.

1

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .