Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

How to resume securely and reliably process of copying file $A into backup location $B done with pv "$A" > "$B" or cat "$A" > "$B" ?

(let's assume file $A is very big, e.g. LVM2 snapshot file)

Is it achievable with dd ?

Preffered: bash or python (preferably python3) solutions.

Example scenario: pv "$A" > "$B" interrupted after copying 90%. How to resume it, in order to finish copying process but not repeating whole work again ?

share|improve this question
    
add comment

3 Answers

up vote 6 down vote accepted

Yes you can use dd to skip the blocks.

A="file1"
B="file2"

BLOCKSIZE=512  # default bs for dd

size_b=$(stat -c "%s" "$B")
skip_blocks=$((size_b / BLOCKSIZE))

dd if="$A" of="$B" skip=$skip_blocks seek=$skip_blocks bs=$BLOCKSIZE

The important parameters here are skip as well as seek:

  • skip: skip BLOCKS ibs-sized blocks at start of input
  • seek: skip BLOCKS obs-sized blocks at start of output
share|improve this answer
add comment

You want rsync:

rsync -a --append "$A" "$B"
share|improve this answer
    
I do not want to use rsync. Let's assume A is 1TB and I've copied already 900GB of data and there are 100G remaining. rsync would read whole 1TB, while I need only the last 100G ! -> Read first about algorithm which is used: en.wikipedia.org/wiki/Rsync#Algorithm –  Grzegorz Wierzowiecki Jul 28 '12 at 18:23
    
@GrzegorzWierzowiecki I think you're wrong. Unless my experience with rsync is entirely wrong, rsync will read what it needs to read to verify the output is right, until it hits a point where there start to appear disparities, and then it will continue from that marked point. This looks to be pretty-much exactly what is needed. –  killermist Jul 28 '12 at 23:35
    
For this to work you also need to add --append. –  Thor Jul 29 '12 at 1:06
    
Thanks @Thor for stressing this option. As I've double checked it seems that behaviour of --append has changed since version 3.0.0. Could you ensure me if currently --append ignores already copied part, while --append-verify reads whole A for checksum check ? (before version 3.0.0. --append behaved like --append-verify -> That's the reason of misunderstanding) –  Grzegorz Wierzowiecki Jul 29 '12 at 9:54
3  
@GrzegorzWierzowiecki: I hadn't noticed this change. I just tested it, and indeed --append blindly appends to the file. --append-verify does the same but runs checksums at the end, if the checksums do not match rsync seems to do a new copy. –  Thor Jul 29 '12 at 10:36
add comment

Did you try dd skip with an offset of B's real file size (independent of the partition block size)?

That would get you the missing part. At that point you could directly cat them together into a new file with cat "$B" "$A2" >> "$C"; #mv "$C" "$B" (where $C is of course the missing part on a path with enough space).

cat works fine for concatenating binaries too and in this case you do not have multiple file headers that would precent simple scripted merges. There's a chance the end of $B is corrupted, but in that case you could cut it short and reread more of $A in the initial dd step.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.