9

How to resume securely and reliably process of copying file $A into backup location $B done with pv "$A" > "$B" or cat "$A" > "$B" ?

(let's assume file $A is very big, e.g. LVM2 snapshot file)

Is it achievable with dd ?

Preffered: bash or python (preferably python3) solutions.

Example scenario: pv "$A" > "$B" interrupted after copying 90%. How to resume it, in order to finish copying process but not repeating whole work again ?

Improve this question
1

3 Answers 3

Reset to default
13

Yes you can use dd to skip the blocks.

A="file1"
B="file2"

BLOCKSIZE=512  # default bs for dd

size_b=$(stat -c "%s" "$B")
skip_blocks=$((size_b / BLOCKSIZE))

dd if="$A" of="$B" skip=$skip_blocks seek=$skip_blocks bs=$BLOCKSIZE

The important parameters here are skip as well as seek:

  • skip: skip BLOCKS ibs-sized blocks at start of input
  • seek: skip BLOCKS obs-sized blocks at start of output
Improve this answer
10

You want rsync:

rsync -a --append "$A" "$B"
Improve this answer
5
  • 1
    I do not want to use rsync. Let's assume A is 1TB and I've copied already 900GB of data and there are 100G remaining. rsync would read whole 1TB, while I need only the last 100G ! -> Read first about algorithm which is used: en.wikipedia.org/wiki/Rsync#Algorithm
    – Grzegorz Wierzowiecki
    Commented Jul 28, 2012 at 18:23
  • 1
    @GrzegorzWierzowiecki I think you're wrong. Unless my experience with rsync is entirely wrong, rsync will read what it needs to read to verify the output is right, until it hits a point where there start to appear disparities, and then it will continue from that marked point. This looks to be pretty-much exactly what is needed.
    – killermist
    Commented Jul 28, 2012 at 23:35
  • 1
    For this to work you also need to add --append.
    – Thor
    Commented Jul 29, 2012 at 1:06
  • Thanks @Thor for stressing this option. As I've double checked it seems that behaviour of --append has changed since version 3.0.0. Could you ensure me if currently --append ignores already copied part, while --append-verify reads whole A for checksum check ? (before version 3.0.0. --append behaved like --append-verify -> That's the reason of misunderstanding)
    – Grzegorz Wierzowiecki
    Commented Jul 29, 2012 at 9:54
  • 5
    @GrzegorzWierzowiecki: I hadn't noticed this change. I just tested it, and indeed --append blindly appends to the file. --append-verify does the same but runs checksums at the end, if the checksums do not match rsync seems to do a new copy.
    – Thor
    Commented Jul 29, 2012 at 10:36
3

Did you try dd skip with an offset of B's real file size (independent of the partition block size)?

That would get you the missing part. At that point you could directly cat them together into a new file with cat "$B" "$A2" >> "$C"; #mv "$C" "$B" (where $C is of course the missing part on a path with enough space).

cat works fine for concatenating binaries too and in this case you do not have multiple file headers that would precent simple scripted merges. There's a chance the end of $B is corrupted, but in that case you could cut it short and reread more of $A in the initial dd step.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .