Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

On occasion I've seen comments online along the lines of "make sure you set 'bs=' because the default value will take too long," and my own extremely-unscientific experiences of, "well that seemed to take longer than that other time last week" seem to bear that out. So whenever I use 'dd' (typically in the 1-2GB range) I make sure to specify the bytes parameter. About half the time I use the value specified in whatever online guide I'm copying from; the rest of the time I'll pick some number that makes sense from the 'fdisk -l' listing for what I assume is the slower media (e.g. the SD card I'm writing to).

For a given situation (media type, bus sizes, or whatever else matters), is there a way to determine a "best" value? Is it easy to determine? If not, is there an easy way to get 90-95% of the way there? Or is "just pick something bigger than 512" even the correct answer?

I've thought of trying the experiment myself, but (in addition to being a lot of work) I'm not sure what factors impact the answer, so I don't know how to design a good experiment.

share|improve this question

4 Answers 4

up vote 9 down vote accepted

dd dates from back when it was needed to translate old IBM mainframe tapes, and the block size had to match the one used to write the tape or data blocks would be skipped or truncated. (9-track tapes were finicky. Be glad they're long dead.) These days, the block size should be a multiple of the device sector size (usually 4KB, but on very recent disks may be much larger and on very small thumb drives may be smaller, but 4KB is a reasonable middle ground regardless) and the larger the better for performance. I often use 1MB block sizes with hard drives. (We have a lot more memory to throw around these days too.)

share|improve this answer
    
Hard drives or USB mass storage devices are either 512 or 4096 (newer) bytes. Optical and direct access flash media is 2048 bytes. Can't go wrong with 4096 bytes. –  ultrasawblade Mar 17 '11 at 13:34
2  
Why the copying program's block size should have anything to do with the underlying device's characteristics (tapes excepted)? The kernel does its own buffering (and sometimes prefetching) anyway. –  Gilles Mar 17 '11 at 22:43
    
To minimize fractional buffers; things in general go faster when you use aligned buffers because the kernel can start buffer reads/writes at sector (or better, track or cylinder, but I think modern drives lie about those) and kernel buffer boundaries, because the kernel isn't having to skip over stuff or read extra stuff or manage partial buffers. Certainly you can just let the kernel deal with it all, but if you're copying gigabytes of data that extra work can cut the copy time down considerably. –  geekosaur Mar 17 '11 at 22:53
    
You (generally) need to include @Gilles if you want me to be notified of your comment reply, see How do comment @replies work?. Since I happened to be passing by: the kernel will deal with it all anyway. Your claim that “that extra work can cut the copy time down considerably” doesn't agree with my benchmarks, but different systems may have different behaviors, so please contribute timings too! –  Gilles Mar 17 '11 at 23:07
    
@Gilles: sorry, I had mistaken you for the original asker. –  geekosaur Mar 17 '11 at 23:09

There's but one way to determine the optimal block size, and that's a benchmark. I've just made a quick benchmark. The test machine is a PC running Debian GNU/Linux, with kernel 2.6.32 and coreutils 8.5. Both filesystems involved are ext3 on LVM volumes on a hard disk partition. The source file is 2GB (2040000kB to be precise). Caching and buffering are enabled. Before each run, I emptied the cache with sync; echo 1 >|/proc/sys/vm/drop_caches. The run times do not include a final sync to flush the buffers; the final sync takes on the order of 1 second. The same runs were copies on the same filesystem; the diff runs were copies to a filesystem on a different hard disk. For consistency, the times reported are the wall clock times obtained with the time utility, in seconds. I only ran each command once, so I don't know how much variance there is in the timing.

             same   diff
dd bs=64M    71.1   51.3
dd bs=1M     73.9   41.8
dd bs=4k     79.6   48.5
dd bs=512    85.3   48.9
cat          76.2   41.7
cp           77.8   45.3

Conclusion: a large block size (several megabytes) helps, but not dramatically (a lot less than I expected for same-drive copies). And cat and cp don't perform so badly. With these numbers, I don't find dd worth bothering with. Go with cat!

share|improve this answer
    
I'd recommend the OP to do his own benchmarking, but anyway, nice answer! –  ninjalj Mar 17 '11 at 23:00
    
@ninjalj I could do similar benchmarking but are the results still relevant when I write to an SD card through my internal reader? Through an external reader? To a USB stick? To some media through my ExpressCard interface? Will there be a difference between a 512MB SD card and an 8GB SD card? Does the destination or source filesystem type have an effect, so I need to re-run all my tests on my ext4 work laptop? There are too many variables and I don't know which ones are relevant! –  drewbenn Mar 18 '11 at 2:58
    
not in the context of the topic, @Gilles, what is the difference in echo 1 >|/proc/sys/vm/drop_caches and echo 1 > /proc/sys/vm/drop_caches –  Nikhil Mulley Dec 23 '11 at 14:09
    
@Nikhil >| is the same as > except that under set -o noclobber, the shell will complain that the file exists if you use >. –  Gilles Dec 23 '11 at 21:29

I agree with geekosaur that the size should be a multiple of the block size, which is often 4K.

If you want to find the block size stat -c "%o" filename is probably the easiest option.

But say you do dd bs=4K, that means it does read(4096); write(4096); read(4096); write(4096)...

Each system call involves a context switch, which involves some overhead, and depending on the I/O scheduler, reads with interspersed writes could cause the disk to do lots of seeks. (Probably not a major issue with the Linux scheduler, but nonetheless something to think about.)

So if you do bs=8K, you allow the disk to read two blocks at a time, which are probably close together on the disk, before seeking somewhere else to do the write (or to service I/O for another process).

By that logic, bs=16K is even better, etc.

So what I'd like to know is if there is an upper limit where performance starts to get worse, or if it's only bounded by memory.

share|improve this answer
    
Profile, don't speculate! –  Gilles Mar 17 '11 at 23:30
    
Absolutely. I am doing that now. But your benchmarks already illustrate the point. –  Mikel Mar 17 '11 at 23:38
1  
The Linux Programming Interface agrees with me. See Chapter 13 - File I/O Buffering. –  Mikel Mar 18 '11 at 1:13
3  
Interestingly, their benchmarks suggest there is little benefit above 4K however. –  Mikel Mar 18 '11 at 1:24
1  
Also, apparently the default file read ahead window is 128 KB, so that value might be beneficial. –  Mikel Mar 18 '11 at 1:33

As Gilles says, you can determine the optimal parameter for the bs option to dd by benchmarking. This, though, begs the question: how can you conveniently benchmark this parameter?

My tentative answer to this question is: use dd-opt, the utility I've recently started working on to solve precisely this problem :)

share|improve this answer
1  
Very nice and interesting utility ;) –  VitoShadow Nov 10 '13 at 10:55

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.