1

I have some tape drives and decided to back my data to tape (one of the reasons is that most of my data is not changing, so once I do a full backup, the tapes will have better chance of working after sitting on a shelf for a while than hard drives).

I would like to use simple Linux tools to do those backups, not some more complicated software like bacula that have its own database. My use case is a bit different and I would rather write my own scripts to do this.

tar and cpio are useful for this, but, because of the format, it needs to read the entire tape (possibly taking hours) to list the files and if I want to restore a single file at the end of the tape, it needs to read the whole tape again.

Due to the way LTO tapes are written (multiple passes to fill the tape), it is possible to seek to a file at the end of the archive relatively quickly without reading all the data from the beginning of the tape.

For example, using mt to get to the next filemark takes much less time than tar to scan the archive to produce the file list or to restore a 1KB file from the end of the archive.

It also should be possible to put a "table of contents" at the beginning of the tape so that you know what files are on the tape and where they are.

Software like BackupExec can do this - it only takes a short time to "catalog" a tape and also takes a short time to restore a file that is at the end of the tape.

I could run ls (or find) on the directory, write the output to tape and then tar the directory to tape. This would solve the listing problem, but tar would still take forever to restore the last file.

Is there a simple tool that is similar to tar, but can create the file list at the beginning of the archive and be able to seek to restore a file?

7
  • perhaps use genisofs instead of tar
    – Jasen
    Jan 26, 2020 at 8:44
  • genisofs was made by debian to create defective fileystem images. It is a bad choice in general and useless in ihis case in special, since it writes in variyng block sizes. The latter applies to its bug-free counterpart mkisofs as well.
    – schily
    Jan 26, 2020 at 9:34
  • 1
    If you want to go down the "keep it really simple" route, I'm not sure what more you want than mt and tar -B on the tape's non-rewind device /dev/nst0. Remember that whenever you close the device you get an EOF mark so you can't update in situ, only append.
    – roaima
    Jan 26, 2020 at 12:35
  • 1
    @schily looks like the -B has different meanings between Solaris tar and GNU tar. Or I could be plain wrong. It's been a (very) long time since I last used tape.
    – roaima
    Jan 26, 2020 at 13:35
  • 1
    @roaima Try to start with reading both man pages... I see no difference. BTW: This is not a Solaris invented option but exists since the very beginning of TAR in the 1970s. In general, it is however unfortunate that GNU tar introduced many options that are in conflict with older options of the same name from SunOS tar or star , but not in this case.
    – schily
    Jan 26, 2020 at 13:49

2 Answers 2

1

If you can skip up the tape at will, I don't see a significant advantage in keeping the list of contents at the front anyway.

I would append a number of smaller tar archives to each tape to improve the retrieval time for specific files. Possibly 20 archive subfiles per tape would be a suitable balance.

I would keep the catalogue for each tape on disk: perhaps a subdirectory for each tape, containing a tar -tvf for each tarfile, named like:

tar_yyyymmdd_hhmmss_LTO_nnnnn_sub_vv.toc

That gives you an online search tool for the whole collection, rather than having to know which tape to load to get the list of contents for itself.

I would archive the specific catalogue to the end of its own tape when you were not going to append further, and archive all the catalogues to a separate tape periodically too. It would not hurt to archive the catalogue to its own tape multiple times: you can always find the last version by skip to end of media, then skip back 1.

1
  • Your answer looks like another question. Things are not as simple as you might think. A tar archive is internally structured in units of 512 bytes and written to tape in larger tape block units. In addition, seeking the tape only makes sense for larger distances, while highly optimized TAR implementations like star already introfuced a FIFO 30 years ago and work better for files of small and medium size. Seeking the tape mainly makes sense if you like to speed up the work to approach the first file on interest on a tape.
    – schily
    Jan 26, 2020 at 14:09
0

We used to store multiple tar archives on a single tape, on Solaris with QIC tapes.

To manage the physical tape, there is the mt(1) command. Specifically, this lets you space forward and back using tape marks, using either absolute or relative numbering. Confusingly, the terminology uses "files" in the sense of multiple sub-files separated by tape marks. A complete tar archive would correspond to a "file".

The mt command has a binary and a man page on my Linux Mint 18.1. Tapes are pretty non-standard -- some types of deck won't have all the commands, but tapemarks are pretty fundamental.

Standard tape devices usually rewind by default before and after each use, thereby destroying any pre-positioning you do. Typically, each deck has a designator like /dev/rmt0, and an additional device for the same physical unit like /dev/nrmt0 where the n says "no rewind".

So you embed your append tar command in a script that does something like:

mt -f /dev/nrmt0 rewind
mt -f /dev/nrmt0 eod
tar -f /dev/nrmt0 ...
mt -f /dev/rmt0 offline

You would need to keep a catalogue of which archives are on which tape and which subfile, and your retrieval would be like:

mt -f /dev/nrmt0 rewind
mt -f /dev/nrmt0 fsf 17
tar -f /dev/nrmt0 ...

Breaking your archives into many smaller sections, and skipping between them, would be a significant optimisation for retrieving small numbers of files.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.