Resuming a DD of an entire disk

Question

I'm overwriting my hard drive with random data using the good old dd:

dd if=/dev/urandom of=/dev/disk/by-uuid/etc bs=512

It's a 2TB array and my MacBook (running Linux, ok?) can only write data at around 3.7MB/s, which is pretty pathetic as I've seen my desktop at home do 20MB/s. When I go home tonight, I'd like to stop the dd run here, take it home, and see what kind of progress can be made overnight with a more powerful machine.

I've been monitoring the progress using a simple loop:

while true; do kill -USR1 $PID ; sleep 10 ; done

The output looks like this:

464938971+7 records in
464938971+7 records out
238048755782 bytes (238 GB) copied, 64559.6 s, 3.7 MB/s

If I were to resume the dd pass at home, how would I restart it? I'm aware of the seek parameter, but what do I point it to, the record number or the byte count?

Answer 1

As @don_crissti already commented, just use seek= to resume.

dd if=/dev/urandom of=/dev/disk/by-uuid/etc bs=512 seek=464938971

GNU dd also supports seeking in bytes, so you can resume exactly, regardless of blocksize:

dd if=/dev/urandom of=/dev/disk/by-uuid/etc bs=1M \
   seek=238048755782 oflag=seek_bytes

A larger blocksize should help with speeds even for a slow device like /dev/urandom.

If you are looking for faster alternatives, you could cryptsetup plainOpen with a random key and zero that, it should beat /dev/urandom by an order of magnitude (without AES-NI) or even run at full speed (with AES-NI).

You could also use shred -n 1 if pseudorandom data is good enough for your use case. shred should be able to utilize the full disk speed, even on a very slow machine.

Answer 2

Just a reminder for people who would like to copy rather than just randomizing disks (which is not that common) : you can use skip=BLOCKS to start reading at the proper position, and seek=BLOCKS to start writing at the correct position. Both options use blocks, not bytes. When breaking/restarting, it's advisable to remove a bunch of blocks just in case. It is usually worth raising the bs value above 512, as you can reach better performance if you read a lot of data in a row.

In your case, it is indeed a block value that you need to pass to seek. Maybe you should try to adjust bs to see if you can enhance speed, as /dev/random should go fast (pseudo-random and non-blocking when it has no entropy available)

Answer 3

Cloning a disk:

Expanding on this answer from this thread, this is how one might go about cloning a whole disk and resuming:

This example is optimized for copying from a 5400rpm rotary drive to an SSD on a specific system. gdd represents GNU dd:

> sudo gdd 'if=/dev/rdisk3' 'of=/dev/rdisk6' bs=4M status=progress
247426187264 bytes (247 GB, 230 GiB) copied, 2082 s, 119 MB/s
59012+0 records in
59011+0 records out
247510073344 bytes (248 GB, 231 GiB) copied, 2082.92 s, 119 MB/s

I can resume this in one of two ways:

> sudo gdd 'if=/dev/rdisk3' 'of=/dev/rdisk6' \
bs=4M \
seek=59011 skip=59011 \
status=progress

Or:

> sudo gdd 'if=/dev/rdisk3' 'of=/dev/rdisk6' \
bs=4M \
seek=247510073344 skip=247510073344 \
oflag=seek_bytes iflag=skip_bytes \
status=progress

In the first example, the reason we use 59011 and not 59012, is because 59011 is how many block sized records were fully copied before being interrupted. (records out).

Answer 4

dd with a tiny block size like 512 bytes is likely to be a lot slower than your disk's maximum throughput. Use a higher block size (on a hunch I'd say a few MB) for good performance. Or use cat — on Linux I found cat to be as fast as dd with the optimal block size when a single disk was involved (I don't know if that holds for OSX as well).

To find how far cat has reached, run lsof -p1234 where 1234 is the process ID of the cat process.

To resume from a position, use

{ dd bs=1 seek=123456; cat /dev/urandom; } >/dev/disk/…

where 123456 is the offset in bytes.

Answer 5

If you can, use

#ddrescue from to logfile
ddrescue /dev/sda /dev/sdb logfile

Keep the logfile off either the from or the to... and if you have to hit ctrl-c or something, ddrescue will read its own logfile and automatically resume. It also copies everything easy first, and since it can resume from the logfile, you can tell it to "go back and try harder" for the others.