4

I'm archiving a few directories every night to LTO-7 tape with about 100 or so large (2GB) files in each of them.

As a check that the data has been written correctly, I'm verifying that the number of bytes reported written is the same as what should have been written.

I'm first looking at the size of the archive by doing a tar dry-run:

tar -cP --warning=no-file-changed $OLDEST_DIR | wc -c

Then I'm creating the archive with:

tar -cvf /dev/nst0 --warning=no-file-changed --totals $OLDEST_DIR

If the filesizes match, then I delete the original file.

The problem is that the dry-run has to read the entire contents of the files and can take several hours. Ideally, it should use the reported filesizes, apply the necessary padding / aligning, and report back the size rather than thrashing the disk for hours.

Using du -s or similar doesn't work because the sizes don't quite match (filesystems treat a directory as 4096 bytes, tar treats it as 0 bytes for example).

Alternatively, is there a better way of checking that the file has been correctly written? I can't trust tar's return code, since I'm ignoring certain warnings (to handle some sort of bug with tar/mdraid)

Improve this question
1
  • Answering last question: the best way to check if files are written correctly is to restore them on different place and compare the content :(
    – Romeo Ninov
    Commented Sep 7, 2016 at 15:10

3 Answers 3

Reset to default
3

I agree it would be wise to use proper tape archiving software, but failing that you should simply pipe your real tar output into a checksumming program and then onto your tape. For example,

tar -cPf - --warning=no-file-changed $OLDEST_DIR |
tee >(md5sum > sum.md5) |
dd of=/dev/nst0

You can then re-read the tape to compare checksums. 

dd if=/dev/nst0 | md5sum | diff - sum.md5

You may need to add appropriate options to dd for your device. I used to add the checksum value itself onto the end of the tape so it can be checked again at any time. You also need to check the initial tar for errors, of course.

Improve this answer
1
  • You can use a variant of meuh's command to simply record the byte size as emitted by tar, before it gets to the tape: ` tar -cPf - --warning=no-file-changed $OLDEST_DIR | tee >(wc -c >/tmp/size) dd of=/dev/nst0 bs=32k ` However, overall I agree you should look at a more tape-aware backup app, like Amanda, Bacula or LTFS.
    – robbat2
    Commented Mar 25, 2017 at 6:13
2

If you add an extra v to your tar command which writes to the drive, it will report the file sizes; you could perhaps parse that and compare the file sizes, without having to read all the files twice.

You have to realise that this is no substitute for proper verification, and the only real verification of a backup is a restore... Note that LTO drives verify writes as they go, so you're not quite driving blind here. But simply relying on file size comparisons doesn't tell you all that much!

I'd actually strongly recommend using proper backup software, such as Bacula which is ideally suited for tape backups. Once you've set it up, it will take care of your verifications for you.

Improve this answer
1

to calculate the size of a tar without creating it, with the GNU implementation of tar, use:

     tar --totals -cf /dev/null folder/
time tar --totals -cf /dev/null folder/  # fast!
Improve this answer
3
  • Would it not need to still read all data, which the user in the question said could take hours?
    – Kusalananda ♦
    Commented Jun 12 at 12:22
  • The OP never used /dev/null in his supposed dry-run command in the original top post. (Maybe that was his mistake?) If I run the command on a 2.8GB file and it takes 0.009s to report the size (so basically instantaneous)
    – ajneu
    Commented Jun 12 at 12:47
  • 1
    So if you use -c -f /dev/null, GNU tar only stats the files and will not read all data? How does that work with sparse files?
    – Kusalananda ♦
    Commented Jun 12 at 12:51

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .