Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I like to clone a whole partition or a whole hard drive onto a larger external disk but like to create a sparse file. I often use dd for cloning, but it doesn't support sparse files. As a workaround I used something like:

cp --sparse=always <(dd if=/dev/sda1 bs=8M) /mount/external/backup/sda1.raw

However this is a little too tricky for my taste and doesn't allow me to resume the process if aborted. It is funny that there is a NTFS tool for this (ntfsclone) but no such tool exists for the native file systems of Linux (EXT2-4).

Is there some better tool for this, e.g. a dd variant with sparse support? I do not look for some proprietary software for disk backups but simply want to make a sparse clone copy which I can mount as loop device if required.

share|improve this question
5  
+1 for creative use of cp, it never occurred to me that you could sparse-copy a disk image. I always just compressed them if I needed to save space. Now why is that in a question not an answer? –  Caleb Jul 20 '11 at 20:05

4 Answers 4

up vote 13 down vote accepted

You want dd_rescue.

dd_rescue -a -b 8M /dev/sda1 /mount/external/backup/sda1.raw
share|improve this answer
1  
Great! The manual says "If the copying process is interrupted by the user it is possible to continue at any position later." and "-a spArse file writing (default=no)". Exactly what I want! Thanks! –  Martin Scharrer Jul 24 '11 at 21:56
2  
Looking for dd_rescue online I found out that there is also a different tool called ddrescue (without the underscore) which was developed independently from dd_rescue but seems to do basically the same. I just mention that here as a general FYI. –  Martin Scharrer Jul 24 '11 at 22:11
    
Yeah, dd_rescue and ddrescue aren't the same thing. Theoretically they do the same job, but generally I've had better luck with the older/original dd_rescue. –  Steven Pritchard Jul 24 '11 at 22:49
1  
In case anyone is wondering, you can stop the copy at any time with Ctrl-C. It will show you your current position, and you can use that value to restart by adding -s and the position to the original command. (So it would look like dd_rescue -a -b 8M -s 42000k /dev/sda1 /mount/external/backup/sda1.raw.) –  Steven Pritchard Jul 24 '11 at 23:05
    
@Steven Pritchard: No need to remember the position. Specify a third filename, which will be the logfile, and on restart it'll read that and pick up where it left off. –  Tanith Rosenbaum Nov 9 at 19:38

Just for completeness the call for ddrescue:

$ ddrescue -S -b8M /dev/sda1 /mount/external/backup/sda1.raw

Or with long option:

$ ddrescue --sparse --block-size 8M /dev/sda1 /mount/external/backup/sda1.raw

Or if you prefer MiBs:

$ ddrescue -S -b8Mi /dev/sda1 /mount/external/backup/sda1.raw

Note that GNU ddrescue and dd_rescue are different programs. But GNU ddrescue seems to be more widespread. For example it is already packaged with GRML.

share|improve this answer

There was a patch offered in 2007 to provide sparse file support in GNU dd, but it looks to have not made it into coreutils (at least not as of 8.4). I doubt dd has changed too much since then, the patch might apply against the current version without a lot of work.

I'm also really impressed by the creative use of cp in your question, and it got me on the track of using it to accomplish resuming (here resuming from ~80M into the source):

cp --sparse=always \
  <(dd if=/dev/sda1 bs=8M skip=10) /dev/stdout \
  | dd bs=8M seek=10 of=/mount/external/backup/sda1.raw

Edit: scratch that. The second dd would of course be seeking to the wrong position in the output file, since it's not the same length as the input.

share|improve this answer
    
As is the case with bhinesley's answer, it would be best to log dd's progress for accurate resuming. If you were to use this approach for both the first run and resumes, and log both parallel dd's independently, then you could know how far into the output to seek. If I have time I'll try to work this up. –  Eli Heady Jul 22 '11 at 18:48
1  
Thanks for the link to the patch. I was starting to think about programming something like it by myself :-) Sparse files can't be pipped so your code won't work. –  Martin Scharrer Jul 22 '11 at 18:50
    
Yup, I just discovered that myself. Oh well, it was fun finding crazy new uses of cp - thanks! –  Eli Heady Jul 22 '11 at 18:58

Note: this doesn't work for the reasons described in the comments, I'm leaving it here for reference.

Monitor the statistics of dd by using kill -USR1:

$ cp --sparse=always <(dd if=/dev/urandom bs=8M) \
    /mount/external/backup/sda1.raw&
$ watch kill -USR1 `pidof -s /bin/dd`

Resume by using skip/seek:

$ i_bytes= # get from the last dd statistic
$ o_bytes=`du -b /mount/external/backup/sda1.raw | cut -f 1`   
$ cp --sparse=always <(dd if=/dev/urandom bs=8M skip=$i_bytes \
    seek=$o_bytes) /mount/external/backup/sda1.raw&
$ watch kill -USR1 `pidof -s /bin/dd`

Without $i_bytes it would be more difficult to resume. It's probably easiest to log the dd statistics to a file in case the machine crashes or whatever.

share|improve this answer
    
The second cp will just overwrite the existing file, wouldn't it? I know the skip and other options of dd. The issue is the cp part. –  Martin Scharrer Jul 21 '11 at 18:14
    
Hm, yeah, you're right. –  bhinesley Jul 21 '11 at 18:18

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.