4

I've a need to archive some big directory structures on an NFS server. They're not likely to be needed particularly soon, if ever - they're just being retained for policy reasons.

To this end, I'm making tarballs, and probably eventually writing them out to tape.

There's just one problem - I'm having a bit of difficult with the really large volumes (10TB+) - the runtime is sufficiently long that it gets left overnight, and in a few cases it seems to have 'stalled' - it's not too easy to tell for a backgrounded tar xvfz.

And then things like running out of space, network interruption etc. means that for things that don't complete in a single session, I'm not entirely sure that the archive is a) complete and b) entirely valid.

So hoping for some advice - ideally what I'd like is something resumable, like rsync, which I can multi-pass the copy, without starting over.

Is there a way to "rsync to a tar.gz"? A not-too-expensive way of verifying file writes? I'm currently looking at 'extract, shasum and compare' but that's also a rather expensive/intensive process.

Improve this question

2 Answers 2

Reset to default
2

Perhaps splitting your backups would be a step towards addressing your problem?

tar cvzf - /your/dir/ | split --bytes=1000MB - backup.tar.gz

Or you can look into dar, perhaps. It has splitting built-in: http://dar.linux.free.fr/doc/Features.html

I also found some info on how to resume an interrupted dar backup job that might help:

http://sourceforge.net/p/dar/mailman/message/30863378/

Improve this answer
1
  • 1
    Could you include the relevant info about resuming in your question please? That link might break, making your answer far less useful.
    – terdon ♦
    Commented Oct 21, 2015 at 11:45
1

To verify that the operation was successful, just add && echo something to each tar operation:

tar cvzf target.tar "$source" && echo "$source tarred successfully" >> progress.log

It won't help you resume, but at least, that way you'll know whether your tar worked.

Another thing that might help would be to first gzip and then tar:

find "$source" -type f -exec gzip {} + && tar cvf "$source".tgz "$source" && 
    echo "$source tarred successfully" >> progress.log

That way, you can easily check whether a file was compressed or not. The tar operation is relatively cheap, it's the compression that slows it down. By decoupling the two, you'll be able to get more fine grained control.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .