34

ORIGINAL QUESTION:

If I have 2 identical hard drives with the following characteristics:

  • SATA 6.0 Gb/s
  • 5400 rpm
  • 3TB

How long should a full dd copy take to complete?

So far it's been running for 5 hours and still going...

I am using Linux Ubuntu 12.04 64bit and the command I am using is:

dd if=/dev/sdb of=/dev/sdc

UPDATE: 1

I can now see the progress, and it's been 6+ hours for copy 430GB. The HDD is 3TB...

Is there no faster way of doing this?


UPDATE: 2

This seems a lot better than before (Thanks to Groxxda for the suggestions):

sudo dd if=/dev/sdb bs=128K | pv -s 3000G | sudo dd of=/dev/sdc bs=128K

ETA is about 9 hours for 3TB, whereas before it reached 430GB after 6 hours, so I am guessing it would have taken about 36 hours using the previous command.

  • Try to grab the statistics of the process: Sending a USR1 signal to a running 'dd' process makes it print I/O statistics to standard error and then resume copying. $ dd if=/dev/zero of=/dev/null& pid=$! $ kill -USR1 $pid Check your man page for the actual signal as it differs for different dd implementations. – groxxda Jul 12 '14 at 13:20
  • @Groxxda, I have no idea how to do that. – oshirowanen Jul 12 '14 at 13:20
  • 1
    GNU dd uses SIGUSR1, and BSD dd uses SIGINFO – groxxda Jul 12 '14 at 13:22
  • Also what do you mean with "connected to the same sata cable"? Are you using some sort of port multiplier? (If you achieve a transfer rate of 150MB/s it should take you 5-6hrs, but I think half of that is more realistic.) – groxxda Jul 12 '14 at 13:31
  • 1
    You may be able to speed up the process by specifying a different (bigger) blocksize (bs= argument to dd). Also consider connecting each HDD to its own sata port. – groxxda Jul 12 '14 at 13:38
58

dd was useful in the old days when people used tapes (when block sizes mattered) and when simpler tools such as cat might not be binary-safe.

Nowadays, dd if=/dev/sdb of=/dev/sdc is a just complicated, error-prone, slow way of writing cat /dev/sdb >/dev/sdc. While dd still useful for some relatively rare tasks, it is a lot less useful than the number of tutorials mentioning it would let you believe. There is no magic in dd, the magic is all in /dev/sdb.

Your new command sudo dd if=/dev/sdb bs=128K | pv -s 3000G | sudo dd of=/dev/sdc bs=128K is again needlessly slow and complicated. The data is read 128kB at a time (which is better than the dd default of 512B, but not as good as even larger values). It then goes through two pipes before being written.

Use the simpler and faster cat command. (In some benchmarks I made a couple of years ago under Linux, cat was faster than cp for a copy between different disks, and cp was faster than dd with any block size; dd with a large block size was slightly faster when copying onto the same disk.)

cat /dev/sdb >/dev/sdc

If you want to run this command in sudo, you need to make the redirection happen as root:

sudo sh -c 'cat /dev/sdb >/dev/sdc'

If you want a progress report, since you're using Linux, you can easily get one by noting the PID of the cat process (say 1234) and looking at the position of its input (or output) file descriptor.

# cat /proc/1234/fdinfo/0
pos:    64155648 
flags:  0100000

If you want a progress report and your unix variant doesn't provide an easy way to get at a file descriptor positions, you can install and use pv instead of cat.

  • What is strange with large blocks is that the bottleneck is the disk, so what makes cat faster than dd ? Could it be that cat uses the cache ? – Emmanuel Jul 12 '14 at 21:37
  • 1
    @Gilles, thanks for the answer. I have another five 3TB drives to clone and will try the cat option next. As far as I can tell, that new dd command is going to take another 3 hours to complete, to about 11 hours in total. If the cat approach is faster than 11 hours for the second 3TB HDD, I will use that method for the remaining drives. – oshirowanen Jul 12 '14 at 21:44
  • 2
    @Gilles, so to get progress report, do I use sudo sh -c 'pv /dev/sdb >/dev/sdc' instead of sudo sh -c 'cat /dev/sdb >/dev/sdc'? – oshirowanen Jul 13 '14 at 6:46
  • 1
    @oshirowanen Yes, use pv where you'd use cat. – Gilles Jul 13 '14 at 10:50
  • 1
    @unhammer No! You would copy each device one by one, e.g. cat /dev/sdc1 >/dev/sdb1 && cat /dev/sdd1 >/dev/sde1. Concatenating the two parts doesn't make sense. If you want to change the structure of the btrfs volume, to change it from having two subvolumes to one, you need to use btrfs tools to change the structure, or as you did to create a new filesystem with the desired structure and copy the files. – Gilles Sep 22 '17 at 8:44
7

dd uses a very small blocksize by default (512 bytes). That results in a lot of overhead (one read() and write() syscall for every 512 bytes).

It goes a lot faster when you use a larger blocksize. Optimal speeds start at bs=64k or so. Most people use a still larger bs=1M so it becomes human readable (when dd says it copied 1234 blocks, you know it's 1234 MiB without doing any math). Using even larger blocksizes are unlikely to result in speed improvements, just higher memory consumption.

So the command should be:

dd bs=1M if=/dev/sdb of=/dev/sdc

If you already have a slow dd running, you can interrupt it and resume with a faster dd instance. For this it is important to know how far the copy progressed already. dd usually prints the progress when you cancel it, or you can send it the USR1 signal while it is running to make it print its progress.

kill -USR1 $(pidof dd)

For example if it copied more than 1234MiB, you can resume at position 1234MiB using:

dd bs=1M seek=1234 skip=1234 if=/dev/sdb of=/dev/sdc

If it copied fewer than 1234MiB, your copy will be incomplete. If it copied more than 1234MiB, it will re-copy some already copied parts, which normally does not do any harm. So if in doubt you should pick a value slightly smaller than what you believe was already copied.

5

Getting statistics about ongoing dd process

You can use the kill command with the appropriate signal to make dd output statistics to standard error.
From the GNU dd man page:

Sending a USR1 signal to a running 'dd' process makes it print I/O statistics to standard error and then resume copying.
      $ dd if=/dev/zero of=/dev/null& pid=$!
      $ kill -USR1 $pid
      18335302+0 records in 18335302+0 records out 9387674624 bytes (9.4 GB)  copied,  34.6279 seconds, 271 MB/s

Make sure you check your man page for the correct signal first as it may differ on different dd implementations: (BSD dd uses SIGINFO).

Speeding up the process

  1. Connect each HDD to it's own SATA port so the data can be read from one device and written to the other at the same time.
  2. Use an appropriate blocksize using the bs= argument. Have a look at this thread on superuser and try some values for yourself.
  3. Use separate dd invocations for reading and writing and use a pipe to connect them (dd if=/dev/sda bs=1M | dd of=/dev/sdb bs=1M).
    If you do this and specify a blocksize, make sure you use the same blocksize on each invocation.
  4. You may try other optimizations like the direct argument.
  5. Make sure your hard disks are not mounted or it may result in a corrupt copy.
0

Have you tried 'gparted' ? You can literally copy-paste a partition from one drive to another and resize it accordingly as needed. You get transfer rate and time remaining. It uses 'e2image' underneath for linux partitions.

0

There is commercial software called HDClone. It is licensed as both freeware and commercial. Either version can create a bootable pendrive or DVD, as well as copying disks. Connect the source and destination HDs and follow the GUI-based instructions.

The free one will transfer at a rate about 80MB/sec, while the paid versions can go much faster. For NTFS and FAT filesystems, the paid versions can be configured to clone only the occupied bits of the HD, which dramatically increases the speed of each clone.

  • (1) Are you affiliated with this product? (2) Where can we get it? – Scott Oct 9 '17 at 21:10
  • 1
    Or you could just use "cat /dev/sdx > /dev/sdy" for free... – HörmannHH Jul 28 '18 at 13:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.