9

I will be backing up a large (750GB) disk to an external USB disk using dd.
Should I be using redirection or piping? Which is more efficient? Or is there a difference?

Also, what is the best block size? USB is likely to be the bottleneck here.

dd if=/dev/sda bs=1M | gzip -c > /mnt/sdb1/backups/disk.img.gz

gzip -dc /mnt/sdb1/backups/disk.img.gz | dd of=/dev/sda bs=1M

vs

dd if=/dev/sda bs=1M | gzip -c | dd of=/mnt/sdb1/backups/disk.img.gz

dd if=/mnt/sdb1/backups/disk.img.gz | gzip -dc | dd of=/dev/sda bs=1M

Thanks.

migrated from stackoverflow.com Mar 12 '12 at 11:44

This question came from our site for professional and enthusiast programmers.

  • 2
    I would try to backup file systems, not raw disk images. And to tune the parameters, you'll have to benchmark. – Basile Starynkevitch Mar 12 '12 at 6:20
  • dd is archaic (any other tool thankfully does away with bs= and count=, like ddrescue), and in your case also pretty much pointless. Just pipe it into gzip -c </dev/sda >disk.img.gz & gzip -cd disk.img.gz >/dev/sda. – jørgensen Mar 12 '12 at 11:12
  • Or do it with rsync. Just construct the command with the parameters, and it will do you everything in the most efficient way it thinks. – vakufo Mar 12 '12 at 11:53
  • In the second command of the second listing, what is the dd if=.../disk.img.gz | part supposed to do? As far as I understand, gzip -dc (=zcat) doesn't care for its stdin when there's a file argument. – sr_ Mar 12 '12 at 13:57
  • I don't think Linux can reliably reproduce Windows installations by simply copying the file structure. Hence, the need for bit-wise copying like dd. ddrescue looks more robust then plain dd. Would you use GNU ddrescue or dd_rescue? What is a good Live CD to download? – mcu Mar 12 '12 at 20:45
3

Piping involves one more process and one more user-land copy, so it should be more efficient to use redirection.

But I guess that on nowadays hardware & software caching system, it should not make any real difference. Maybe you can have better results using bs=4k or bs=64k, since it's the pipe's limit under linux. See this question for more detail about different bs parameters.

8

You don't need to use dd or piping at all.

</dev/sda gzip >/mnt/sdb1/backups/disk.img.gz
</mnt/sdb1/backups/disk.img.gz gunzip >/dev/sda

I once made a benchmark and found using dd slower than cat for a straight copy between different disks. I would expect the pipe to make any solution involving dd even slower in this case.

  • That is an example of using redirection. – psusi Mar 13 '12 at 17:42
  • @psusi Gah! I meant to write “dd or piping”. Thanks. – Gilles Mar 13 '12 at 17:47
  • Is there a way to combine pv in these commands? – Yaron Feb 19 '18 at 21:24
  • 1
    @Yaron Sure: replace gzip by pv | gzip and gzip -d by gzip -d | gv – Gilles Feb 19 '18 at 21:58
0

Simpler:

gzip -c /dev/sda > disk.img.gz
gzip -dc /mnt/sdb1/backups/disk.img.gz > /dev/sda 

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.