Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

Short version: In what circumstances is dd safe to use for copying data, safe meaning that there is no risk of corruption due to a partial read or write?

Long version — preamble: dd is often used to copy data, especially from or to a device (example). It's sometimes attributed mystical properties of being able to access devices at a lower level than other tools (when in fact it's the device file that's doing the magic) — yet dd if=/dev/sda is the same thing as cat /dev/sda. dd is sometimes thought to be faster, but cat can beat it in practice. Nonetheless, dd has unique properties that make it genuinely useful sometimes.

Problem: dd if=foo of=bar is not, in fact, the same as cat <foo >bar. On most unices¹, dd makes a single call to read(). (I find POSIX fuzzy on what constitutes “reading an input block” in dd.) If read() returns a partial result (which, according to POSIX and other reference documents, it's allowed to unless the implementation documentation says otherwise), a partial block is copied. Exactly the same issue exists for write().

Observations: In practice, I've found that dd can cope with block devices and regular files, but that may just be that I haven't exercised it much. When it comes to pipes, it's not difficult to put dd at fault; for example try this code:

yes | dd of=out bs=1024k count=10

and check the size of the out file (it's likely to be well under 10MB).

Question: In what circumstances is dd safe to use for copying data? In other words, what conditions on the block sizes, on the implementation, on the file types, etc, can ensure that dd will copy all the data?

(GNU dd has a fullblock flag to tell it to call read() or write() in a loop so as to transfer a full block. So dd iflag=fullblock oflag=fullblock is always safe. My question is about the case when these flags (which don't exist on other implementations) are not used.)

¹ I've checked on OpenBSD, GNU coreutils and BusyBox.

share|improve this question
    
I've never seen any Unixy system that really could read a few MiB in a single read(2)... –  vonbrand Jan 23 '13 at 16:14
add comment

2 Answers

The difference between cat <input >output and dd if=input of=output is in the block size they will use. The default for dd if no bs parameter is given is 512 bytes (the size of a disk sector), whereas cat will use whatever blocksize it is coded to work with, which based on a quick inspection of the source code is either 32 KB or the st_blksize value returned by stat() on the input file, whichever is larger. The effect of this can sometimes be seen when reading from a tape drive, where the tape block size is larger than the default used by cat, in which case cat /dev/st0 | <some command> will fail with a data read error, while dd if=/dev/st0 bs=1M | <same command> will work fine.

I can confirm the behaviour you have mentioned with yes, in that dd will copy partial blocks if it cannot read a full one, and the output file ends up at 104 KB in size if the iflag=fullblock is not given, and the expected 10 MB if it is. However, this is only a problem if you are relying on a combination of the bs and count parameters to determine the amount of data copied, which you are in this case. When copying a file or device in its entirety (with no count parameter), it does not matter if partial blocks are copied because the process will only finish when all data has been read.

The conclusion therefore is that dd is always safe to use for copying a whole file or device, because it will read to the end no matter the blocksize, while dd bs=<something> count=<something> may copy less than expected if the source device returns less than the full blocksize in one or more read() calls. This is probably rare when reading from a file or a device like /dev/zero, but according to the man page for read(2) it is theoretically possible during any read call (for example if the process is interrupted by a signal). If copying less than the requested amount of data is unacceptable, the iflag=fullblock parameter can be used to prevent this.

Information about the blocks copied can be seen in the diagnostic output from dd, which lists both the full and partial blocks separately:

$ yes | dd bs=1M count=10 of=outfile
0+10 records in
0+10 records out
180224 bytes (180 kB) copied, 0.00213842 s, 84.3 MB/s

$ yes | dd bs=1M count=10 of=outfile iflag=fullblock
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 0.132989 s, 78.8 MB/s

The block count is listed as <full>+<partial>, so in the first case you can see that 10 partial blocks were copied and the output file contains less data than expected.

share|improve this answer
    
So if you end up with a smaller file than you expected, how could you you tell the difference between dd terminating because of a partial read and the data source actually running out of data? –  Caleb Jul 25 '11 at 10:11
    
The diagnostic output shows info about partial/full blocks, edited answer to include this. –  OrbWeaver Jul 25 '11 at 10:29
1  
The problem with count also occurs with skip (with GNU dd, only if the input isn't seekable). Is dd always safe if neither count or skip are specified? Is dd always safe if skip is specified and the input is seekable? Do all dd implementations write each block fully even if the first write() call is partial? –  Gilles Jul 25 '11 at 11:20
add comment

With sockets, pipes, or ttys, read() and write() can transfer less than the requested size, so when using dd on these, you need the fullblock flag. With regular files and block devices however, there are only two times when they can do a short read/write: when you reach EOF, or if there is an error. This is why older implementations of dd without the fullblock flag were safe to use for disk duplication.

share|improve this answer
    
Is that true of all modern unices? (I know it wasn't true of Linux at some point, possibly up to 2.0.x or 2.2.x. I rembember mke2fs failing silently because it called write() with some non-power-of-2 size (3kB IIRC) and the kernel rounded down to a power of 2.) –  Gilles Jul 25 '11 at 14:47
    
@Gilles that sounds like a different issue entirely. You always have to use a multiple of the proper block size with block devices. I am pretty sure it is true of all unicies, and it is also true for Windows. –  psusi Jul 25 '11 at 14:57
    
Apart from tapes, the block size of a device is purely for the kernel to care about, or not. cat </dev/sda >/dev/sdb works just fine to clone a disk. –  Gilles Jul 25 '11 at 15:00
    
@Gilles that is because cat uses the appropriate block size, as OrbWeaver noted in his answer. –  psusi Jul 25 '11 at 15:51
    
No, there is no “appropriate block size”. cat picks a buffer size for performance; it doesn't get any device-related information from the kernel. Apart from tapes, you can read() and write() to a block device with any size. On Linux at least, st_blksize depends only on the filesystem where the block device inode is located, not on the underlying device. –  Gilles Jul 25 '11 at 17:22
show 1 more comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.