11

I need to copy one disk to another. I tried with the command below and it takes nearly a day to copy 1 TB of disk.

dd if=/dev/sda of=/dev/sdb 

I have tried the same on a Unix system with the command below and it completes within a few hours

dd if=/dev/sda of=/dev/rdsk

What is the alternative I could use to copy from disk to disk as faster?

  • 2
    cp /dev/sda /dev/sdb or (pv /dev/sda > /dev/sdb to get a progress bar) would be a lot quicker. Why would you use dd here? dd would only be useful with things like conv=sync,noerror to handle disks with errors, but even then it would make more sense to use things like ddrescue instead (see also pv's -E option). – Stéphane Chazelas Nov 10 '16 at 9:27
  • 1
    @StéphaneChazelas cat may be even quicker but the difference isn't that dramatic (maybe bigger for device-to-device than file-to-file as in my experiment). – Gilles Nov 10 '16 at 12:44
  • 7
    "I have tried the same on a Unix system" - So on what type of system did you try the first, if not a Unix? Also, what hardware, etc, yaddayadda. – marcelm Nov 10 '16 at 13:13
  • Welcome to dd pitfall #1 – Dmitry Grigoryev Nov 10 '16 at 15:32
  • Used the first in HP-UX (Integrity blade) and earlier used Solaris machine as well. – KKD Nov 11 '16 at 4:15
23

dd has many (weird) options, see dd(1).

You should explicitly state the buffer size, so try

dd if=/dev/sda of=/dev/sdb bs=16M

IIRC, the default buffer size is only 512 bytes. The command above sets it to 16 megabytes. You could try something smaller (e.g. bs=1M) but you should use more than the default (especially on recent disk hardware with sectors of 4Kbytes, i.e. Advanced Format). I naively recommend some power of two which is at least a megabyte.

With the default 512 bytes buffer size, I guess (but I could be very wrong) that the hardware requires the kernel to transfer 4K for each 512 bytes block.

Regarding rdsk, the sd(4) man pages say:

At this time, only block devices are provided. Raw devices have not yet been implemented.

Increase of dd's buffer size will give you more performance for read and write operations. Now all disks have hardware read/write buffer. But if you will increase dd's buffer size more than hardware buffer its performance will decrease because dd will read from first disk to buffer when second disk will have written all from its own hardware buffer. You need set bs option of dd command each time different value for different devices.

  • Whether rdsk is available in Linux systems? I have used in Unix systems. – KKD Nov 10 '16 at 6:02
  • 1
    The page cache will probably deal in 4Kb blocks whatever you do, but you can control how many syscalls dd uses to read that 4Kb. I'm sure there's some read size above which the cost of stalling writes is more expensive than the saved syscalls, but have no idea where the sweet spot is. – Useless Nov 10 '16 at 12:17
  • A block size of a few MB is better than the default 512B but when I benchmarked this I found that cat did just as well (for filesystem-to-filesystem transfer, direct block-to-block may have different performance characteristics). However the difference wasn't dramatic in any case. – Gilles Nov 10 '16 at 12:43
  • I thought using the wrong BlockSize causes corruption if I try to use the new disk with the same filesystem and a different BlockSize – cat Nov 10 '16 at 13:36
  • 1
    Interestingly, in macOS (a SUS-certified, btw) its faster to use /dev/rdiskX as the target when performing dd. – adib Jun 2 at 13:41
13

Years back in Unix-land dd was the required way to copy a block device. That has carried forward as cargo-cult knowledge even though (on Linux-based systems, at least) cat is almost always faster than dd.

However, even back in history a decent block size helped reduce the number of (slow) system calls, given that each system call triggered an I/O operation. The default block size is 512 bytes (one disk sector). Collecting multiple disk blocks together into a single read was - and is - also acceptable. This example uses a 32MB block size:

dd bs=$((512*2048*32)) if=/dev/source of=/dev/target

On current Linux-based systems, though, disks can be most efficiently copied with a simple cat

cat /dev/source >/dev/target

(As noted in the comments on your question pv can be substituted for cat and will give you an indication of progress and throughput.)

  • 2
    Specifically, the reason dd had to be used was a bug in GNU cp and a bug in the linux kernel in the early 90s. The reasons for using dd on historical unix systems was very different, and wanting to copy a whole block device was an unusual thing to want to do. – Random832 Nov 10 '16 at 14:50
  • 1
    @Random832 wanting to copy an entire disk would have been unusual, but I do remember needing to copy partitions around (big ones - 150 or even 200MB) – roaima Nov 10 '16 at 14:55
  • 1
    (The specifics of the bugs: the kernel reported disk usage sizes incorrectly [leading cp to conclude that every source file was a sparse file], and cp did not zero out blocks when copying from a sparse file to a device destination. So any zero block in your source would have whatever garbage happened to already be on the disk) – Random832 Nov 10 '16 at 15:06
6

Very nice question. The raw interface is implemented on some unix systems(tru64,hpux,solaris) but not on linux. The raw interface makes transfer faster because unix I/O is skipped. The block interface(/dev/dsk or /dev/disk) is slower because it uses the unix I/O system. To speedup dd (gnu dd can) use bs=30M or bs=20M depending on your hw. The short answer is : NO it is not implemented,at least as far as I know. I am using linux since the old times of kernel version 2.2 and have never seen rdsk used on unix.

  • 5
    Why do you suggest a block size which is not a power of two? – Basile Starynkevitch Nov 10 '16 at 8:19
  • 1
    @Basile a multiple of the disk block size is sufficient, so 20MiB would be fine. – roaima Nov 10 '16 at 22:15
6

Generally, dd can be avoided in favor of some alternatives. There are several good reasons to use GNU ddrescue instead. In Ubuntu, you can install it with:

sudo apt-get install gddrescue

and just plain ddrescue to use. Note that differently from the package name, the executable does not have the initial g.

Using it is as simple as:

ddrescue inputFile outputFile logFile

The log file (named whatever you choose) lets you pause/stop and restart, without redoing the previous work, which is useful when doing large clones or recovery of disks. By default, it displays progress, current copy speed, average copy speed and number of bad blocks found.

It uses sensible defaults for block size, so copy speed is always as fast as the device can handle, in my experience at least (I've cloned many hundreds of drives with it, all sizes and types).

Often times, drives that are starting to fail have speed issues such as occasional patches of slowness, low average speed, sudden long pauses (bad sectors) or complete resets (severe surface errors). ddrescue can help you identify all the above and restart your clone (provided you specified a log file) even if your drive is resetting itself.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.