3

I'm archiving a few directories every night to LTO-7 tape with about 100 or so large (2GB) files in each of them.

As a check that the data has been written correctly, I'm verifying that the number of bytes reported written is the same as what should have been written.

I'm first looking at the size of the archive by doing a tar dry-run:

tar -cP --warning=no-file-changed $OLDEST_DIR | wc -c

Then I'm creating the archive with:

tar -cvf /dev/nst0 --warning=no-file-changed --totals $OLDEST_DIR

If the filesizes match, then I delete the original file.

The problem is that the dry-run has to read the entire contents of the files and can take several hours. Ideally, it should use the reported filesizes, apply the necessary padding / aligning, and report back the size rather than thrashing the disk for hours.

Using du -s or similar doesn't work because the sizes don't quite match (filesystems treat a directory as 4096 bytes, tar treats it as 0 bytes for example).

Alternatively, is there a better way of checking that the file has been correctly written? I can't trust tar's return code, since I'm ignoring certain warnings (to handle some sort of bug with tar/mdraid)

  • Answering last question: the best way to check if files are written correctly is to restore them on different place and compare the content :( – Romeo Ninov Sep 7 '16 at 15:10
2

If you add an extra v to your tar command which writes to the drive, it will report the file sizes; you could perhaps parse that and compare the file sizes, without having to read all the files twice.

You have to realise that this is no substitute for proper verification, and the only real verification of a backup is a restore... Note that LTO drives verify writes as they go, so you're not quite driving blind here. But simply relying on file size comparisons doesn't tell you all that much!

I'd actually strongly recommend using proper backup software, such as Bacula which is ideally suited for tape backups. Once you've set it up, it will take care of your verifications for you.

2

I agree it would be wise to use proper tape archiving software, but failing that you should simply pipe your real tar output into a checksumming program and then onto your tape. For example,

tar -cPf - --warning=no-file-changed $OLDEST_DIR |
tee >(md5sum > sum.md5) |
dd of=/dev/nst0 

You can then re-read the tape to compare checksums.

dd if=/dev/nst0 | md5sum | diff - sum.md5

You may need to add appropriate options to dd for your device. I used to add the checksum value itself onto the end of the tape so it can be checked again at any time. You also need to check the initial tar for errors, of course.

  • You can use a variant of meuh's command to simply record the byte size as emitted by tar, before it gets to the tape: ` tar -cPf - --warning=no-file-changed $OLDEST_DIR | tee >(wc -c >/tmp/size) dd of=/dev/nst0 bs=32k ` However, overall I agree you should look at a more tape-aware backup app, like Amanda, Bacula or LTFS. – robbat2 Mar 25 '17 at 6:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.