5

I've been using dd in Linux to overwrite an external USB hard drive.

When I use the default block size of 512 bytes, with this command:

# dd if=/dev/zero of=/dev/sdb

throughout the whole operation, the hard drive (/dev/sdb) is being read from and written to alternately, approximately 1GB at a time. I.e. read 1GB...write 1GB...read 1GB...write 1GB etc. As much data is read from the hard drive as is written to it.

I know this is happening because it's showing in my custom Conky panel (diskio_read, diskio_write), which I know to be a 100% reliable indicator of disk I/O activity.

I've repeated this using a different external hard drive on a different computer. It happens via both USB 2.0 and USB 3.0.

In contrast, when I do the same thing, but use a block size of 1MB instead, with this command:

# dd if=/dev/zero of=/dev/sdb bs=1M

apart from a small amount of reading at the start, the hard drive is not read from at all during the operation.

Given that this phenomenon has happened on two different computers and two different hard drives of mine, using a standard Linux distro (Xubuntu 14.04), anyone who wants to should presumably be able to replicate it on their own computer.

Can someone please explain what is happening here?

  • Where do you see this? Please elaborate. – frostschutz Dec 3 '15 at 17:13
  • Where/how do you observe those read/write? (iostat, strace, btrace)? I assume it's Linux, right? What does grep . /sys/block/sdb/queue/*size* give? – Stéphane Chazelas Dec 3 '15 at 17:16
  • I've elaborated on how I know about the I/O. I can't read that file because I'm not overwriting the hard drive any more. – EmmaV Dec 3 '15 at 17:32
  • And this, once again, shows why you should just use cat. – Gilles Dec 3 '15 at 23:45
  • @Gilles My intention was to write nulls through dm-crypt, so I expect there will be the same problem if I use cat. – EmmaV Dec 6 '15 at 0:28
5

If you specify a block size (512 bytes) of less than the block size of the disk (often 4096 bytes, but nowadays maybe more), the block will be partially written, so that the contents of the rest of the block must be preserved before writing.

This is because disk blocks cannot be written to with only 512 bytes, but you have to write a full block at once (4096 or larger). When you write this (4096) amount or more, there is no partial write, so it does not have to read.

0

This is basically a symptom of buffered I/O on a Linux system. A good overview of this in relation to the dd command is located at Is there a way to determine the optimal value for the bs parameter to dd? which supports ctrl-d's answer. However, this also suggests that even larger block sizes (i.e., greater than 1M) might improve performance further.

  • Actually, the reads are not due to buffered I/O. They would be just as necessary if I/O was unbuffered. – Gilles Dec 3 '15 at 23:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.