When is dd suitable for copying data? (or, when are read() and write() partial)

Question

Short version: In what circumstances is dd safe to use for copying data, safe meaning that there is no risk of corruption due to a partial read or write?

Long version — preamble: dd is often used to copy data, especially from or to a device (example). It's sometimes attributed mystical properties of being able to access devices at a lower level than other tools (when in fact it's the device file that's doing the magic) — yet dd if=/dev/sda is the same thing as cat /dev/sda. dd is sometimes thought to be faster, but cat can beat it in practice. Nonetheless, dd has unique properties that make it genuinely useful sometimes.

Problem: dd if=foo of=bar is not, in fact, the same as cat <foo >bar. On most unices¹, dd makes a single call to read(). (I find POSIX fuzzy on what constitutes “reading an input block” in dd.) If read() returns a partial result (which, according to POSIX and other reference documents, it's allowed to unless the implementation documentation says otherwise), a partial block is copied. Exactly the same issue exists for write().

Observations: In practice, I've found that dd can cope with block devices and regular files, but that may just be that I haven't exercised it much. When it comes to pipes, it's not difficult to put dd at fault; for example try this code:

yes | dd of=out bs=1024k count=10

and check the size of the out file (it's likely to be well under 10MB).

Question: In what circumstances is dd safe to use for copying data? In other words, what conditions on the block sizes, on the implementation, on the file types, etc, can ensure that dd will copy all the data?

(GNU dd has a fullblock flag to tell it to call read() or write() in a loop so as to transfer a full block. So dd iflag=fullblock oflag=fullblock is always safe. My question is about the case when these flags (which don't exist on other implementations) are not used.)

¹ _{I've checked on OpenBSD, GNU coreutils and BusyBox.}

Answer 1

The difference between cat <input >output and dd if=input of=output is in the block size they will use. The default for dd if no bs parameter is given is 512 bytes (the size of a disk sector), whereas cat will use whatever blocksize it is coded to work with, which based on a quick inspection of the source code is either 32 KB or the st_blksize value returned by stat() on the input file, whichever is larger. The effect of this can sometimes be seen when reading from a tape drive, where the tape block size is larger than the default used by cat, in which case cat /dev/st0 | <some command> will fail with a data read error, while dd if=/dev/st0 bs=1M | <same command> will work fine.

I can confirm the behaviour you have mentioned with yes, in that dd will copy partial blocks if it cannot read a full one, and the output file ends up at 104 KB in size if the iflag=fullblock is not given, and the expected 10 MB if it is. However, this is only a problem if you are relying on a combination of the bs and count parameters to determine the amount of data copied, which you are in this case. When copying a file or device in its entirety (with no count parameter), it does not matter if partial blocks are copied because the process will only finish when all data has been read.

The conclusion therefore is that dd is always safe to use for copying a whole file or device, because it will read to the end no matter the blocksize, while dd bs=<something> count=<something> may copy less than expected if the source device returns less than the full blocksize in one or more read() calls. This is probably rare when reading from a file or a device like /dev/zero, but according to the man page for read(2) it is theoretically possible during any read call (for example if the process is interrupted by a signal). If copying less than the requested amount of data is unacceptable, the iflag=fullblock parameter can be used to prevent this.

Information about the blocks copied can be seen in the diagnostic output from dd, which lists both the full and partial blocks separately:

$ yes | dd bs=1M count=10 of=outfile
0+10 records in
0+10 records out
180224 bytes (180 kB) copied, 0.00213842 s, 84.3 MB/s

$ yes | dd bs=1M count=10 of=outfile iflag=fullblock
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 0.132989 s, 78.8 MB/s

The block count is listed as <full>+<partial>, so in the first case you can see that 10 partial blocks were copied and the output file contains less data than expected.

Answer 2

With sockets, pipes, or ttys, read() and write() can transfer less than the requested size, so when using dd on these, you need the fullblock flag. With regular files and block devices however, there are only two times when they can do a short read/write: when you reach EOF, or if there is an error. This is why older implementations of dd without the fullblock flag were safe to use for disk duplication.