Resume incomplete disk image with dd over ssh

Question

What I did so far

I wanted to create a disk image of my server HDD partition and copy it to my desktop (client). I used the following command on the server:

dd if=/dev/md3 status=progress bs=500k | bzip2 --best > /mnt/client/image.bz2

where /mnt/client/ is an sshfs mount to my client.

Unfortunately, my client doesn't have a static IP address and the disk image took way longer than expected. At 511 GB (of 1 TB) my ISP forced a new IP address upon me and both ssh connections broke. That is the ssh connection on which dd ... | bzip2 > ... was called and the sshfs mount.

Please note that I need to do a disk image. Copying the files is not enough. And please also note that there is not enough space on my server to store the disk image so it needs to be directly saved to my client.

My questions

How do I resume the disk image?

How do you resume the dd image such that previously copied data doesn't get transferred again? Especially since the already transferred data is inside a compressed archive.

Keeping the compression is preferable since the bandwidth of my network connection seems to be the bottleneck. But it's not strictly necessary if that makes it easier.

For files, I would think of something like rsync but that doesn't apply here.

I thought of decompressing my already copied (partial) archive, measuring its size in bytes, and using something like

dd if=/dev/md3 skip=[size of uncompressed partial image] iflag=skip_bytes

But I'm not sure how I would append the data stream to the existing archive. I figured just using >> to append to the archive probably doesn't work. And also decompressing the archive might take some time which may or may not be necessary. But I don't know how else I would get the uncompressed size of the partial image.

Let's say I ditch the whole compression thing. At least, would dd if=... skip=... >> /mnt/client/uncompressed_image work as intended?

How do I ensure "connection safety" when using ssh?

How do I ensure that my ssh connection doesn't break with a non-static IP address? Or at least how do I ensure that I could easily resume the dd command when it breaks?

In hindsight it might have been smarter to just do something like the following from my client:

ssh root@server "dd if=/dev/md3 | bzip2 --best -c" | dd of=image.bz2

That way I don't need the IP of my client for the sshfs mount. But it will still break the ssh pipe when my IP address changes. So that alone solves nothing.

Answer 1

You've confirmed that you're using a rescue image, so the 1TB source disk is currently unused. That's really good news.

Now, to copy the disk over an unreliable medium you need either a transport layer that can automatically re-establish itself (for example, OpenVPN over UDP), or a means to send restartable "chunks" of data.

Let's assume that everything is controlled from the client, and that you have ssh access mediated with certificates/keys so no passwords are required.

I'm going to suggest that you use 1GB blocks, which means you'll need around 1000 ssh connections to transfer the entire disk. You may want to tweak the blocksize and count near the top of the script. In the section "Valid data" you can choose to write a bzip2 compressed chunk or a normal uncompressed chunk. (bzip2 can deal with multiple compressed chunks, so this isn't an issue.)

#!/bin/bash
#
dev=/dev/sda1   # Device to read
img=image.dat   # Target image filename
bs=32M          # dd blocksize per read
count=32        # Number of blocks per ssh chunk

for (( chunk=0; ; ))
do
    # Grab chunk from server
    #
    ssh -zn root@remoteServer "
        dev=$dev chunk=$chunk bs=$bs count=$count "'
        echo "chunk $chunk from device $dev"
        {
            dd bs=$bs skip=$((chunk*count)) count=$count if=$dev 2>/tmp/dd.$$
            echo $? >/tmp/ss.$$
            dd bs=$bs iflag=fullblock count=$count if=/dev/zero 2>/dev/null
        } | dd bs=$bs iflag=fullblock count=$count 2>/dev/null
        echo "========"
        echo "status $(cat /tmp/ss.$$)"
        cat /tmp/dd.$$
        rm -f /tmp/dd.$$ /tmp/ss.$$
    ' |
        {
            # Extract data from chunk
            #
            IFS= read -r info
            echo "Received: $info"
            dd bs=$bs iflag=fullblock count=$count of=/var/tmp/data.$$
            cat >/var/tmp/meta.$$
        }

    # Append extracted data
    #
    meta=$(cat /var/tmp/meta.$$)
    echo "Meta:"
    echo "$meta" | sed 's/^/| /'

    if [[ "$meta" =~ ([[:digit:]]+)\+[[:digit:]]+' records in' ]]
    then
        # Valid data
        #
        # bzip2 </var/tmp/data.$$ >> "$img.bz2"
        dd bs=$bs count=$count seek=$((chunk*count)) conv=notrunc if=/var/tmp/data.$$ of="$img"

        # Is this all
        #
        if [[ "${BASH_REMATCH[1]}" -lt $count ]]
        then
            # We are done
            #
            break
        fi

        # Next round the loop
        #
        ((chunk++))
    else
        echo "Invalid data received for chunk $chunk; retrying"
    fi
done

# Tidy up
#
rm -f /var/tmp/data.$$ /var/tmp/meta.$$
exit 0

The resulting image will be at least size of the source disk; the extra will be sufficient zero bytes to round up to the next complete chunk size. You could use truncate to reduce the size of the resulting uncompressed image if this is important.

Answer 2

To send the missing bit, you'll still need to recompress from the start I'm afraid, even if you only transfer the missing bits as a compression stream like that of bzip2 depends on what was seen before.

If $s contains the size of what was already transferred, you'd do

dd if=/dev/md3 status=progress bs=500k | bzip2 --best | {
  head -c "$s" > /dev/null
  ssh host 'cat >> /mnt/client/image.bz2'
}

(assuming a head implementation such as GNU head that doesn't read more than requested here).

Or the other way round:

ssh other-host "
  dd if=/dev/md3 status=progress bs=500k | bzip2 --best |
    tail -c +$(($s + 1))" >> /mnt/client/image.bz2

That's assuming /dev/md3 was not modified (not even mounted which could update some "last mount time" field for instance) since last time.

It's also a bad idea to use network file system to transfer large files.