5

I'm doing backups to LTO tape. Because my backups include a lot of small files, which slows down the read from disk. I'm using the buffer command to prevent shoe-shining my tapes:

bkname="test"; tobk="*" 
totalsize=$(du -csb $tobk | tail -1 | cut -f1) 
tar cvf - $tobk | tee >(sha512sum > $bkname.sha512) >(tar -tv > $bkname.lst) | mbuffer -m 4G -P 100% | pv -s $totalsize -w 100 | dd of=/dev/nst0 bs=256k

The problem with this approach is: I can't make a backup spanning multiple tapes, because the tar command isn't directly accessing the tape and therefore won't recognize a full tape.

What would be the correct way to buffer the small files and have multi tape backups at the same time?

8
  • What is the minimum write speed for your drive ? Also have you measured the rate at which tar will output the data on your system (by ending your pipe with `|pv>/dev/null' ? How do they compare ?
    – zeppelin
    Feb 3, 2017 at 19:06
  • Do you know the max capacity of the tape(s) in advance? Would tar options -L (change tape after n kB) and -M (multi volume achieve), or -F (run command at the end of each tape) help? Not sure if you can keep your pipeline as it is, though.
    – dirkt
    Feb 4, 2017 at 11:32
  • @zeppelin: the tape drive supports up to 120MB/s without compression or 240MB/s with compression enabled. tar delivers from 1MB/s up to 200MB/s, depending on the data. A lot of small files are very slow, large files are fast. Feb 5, 2017 at 16:11
  • @dirkt: your suggestions would actually work. The Problem with this approach is, that the tapedrive supports hardware compression. That means it can store 800GB to 1,6TB, depending on the data. So it would be great if tar would recognize a full tape, regardless of the amount of data written. Otherwise I would be wasting space Feb 5, 2017 at 16:11
  • How does the tape drive actually signal a full tape (I don't have the hardware)? Some event on /dev/input/? And if compression is done on-the-fly by writing on /dev/nst0, there's no way to know when the tape will be full beforehand, which is necessary with all this buffering. Is there some way to simulate compression in advance, or compress to a file which can then by written "raw"?
    – dirkt
    Feb 5, 2017 at 16:19

2 Answers 2

2

Consider using star instead of GNU tar. The star program has a buffering tool builtin, solving your problem.

1
1

The size of a tar archive is not the calculated size. Only a preflight run delivers the right size, but it doubles the workload. A example:

~# du -csb /usr | tail -1 | cut -f1
=> 1585916720

~# tar --totals -cf /dev/null /usr
=> 1656514560

My recommendation: Use a powerful backup tool like dar http://dar.linux.free.fr/

2
  • i think dar can't handle tape drives. The progressbar is not that important to me, but thanks for your suggestion! Feb 5, 2017 at 16:14
  • 1
    Tape drive support in dar is the same as in tar; reading and writing to /dev/nst0. It supports volume spanning, but not tape control. This is the job of mt and mtx or a compete backup suite like bareos.
    – ingopingo
    Feb 5, 2017 at 16:27

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .