2

I would like to write a tar archive to tape, and at the same time store the block offsets for each file in a lookup table. Then when I want to restore an individual file from the archive, I would seek to the corresponding block on the tape and restore that file alone.

Does tar support restoring a file in this manner? i.e. when the tape is not positioned at the start of the archive.

Note that I would not use any compression.

If tar doesn't support this how could I accomplish this? (maybe using dd?)

2
  • It would help, if you did write something about the motivation behind this question. So do you have a real issue or did you just missunderstand tar... – schily Jun 4 '20 at 9:47
  • @schily We want to store very large archives (300GB on average) to tape. We would like to be able to restore a single file from the middle of the archive without having to restore the whole archive from tape, or without tar having to scan through the whole archive. We want to do a fast block seek with the tape drive to the location of the file, and then restore the file – swami Jun 4 '20 at 10:15
4

tar does not help you here as you would not know where to seek to, but there is already a similar solution that is in use sice approx. 20 years in the SAMFS/QFS project as a HSM.

In that solution, the starting block numbers are archived in a separate data base.

The problem here is that the tape seek does not work based on 512 byte blocks but rather in a tape record size base which is much larger. So the only way to make this work is to lookup the tape block number for a file, use mt fsr xxx on the tape and then to start tar in ignore checksum errors mode.

This could work, but it has a problem since tar does not exit after it has been able to successfully extract the named file and you need to wait until the current tape file ends.

If you played a bit with this method and believe it is worth, I could add a related option to star to let it exit() after the file has been extracted.

star already has an option -block-number that (together with -v) prints the 512 byte based block numbers when in -c mode. If you divide these numbers by the blocking factor (using integer arithmetic to round down the result), you get the mt fsf parameter you need to seek forward.

The next version of star (in schilytools) will be ready some day in the next week. It will include a new option -one-filé and another option iskip=offset that allows to skip a specified amount of data from the first block in the read data. This will allow you to avoid the -i option.

Since today, schilytools-2020-06-09 is out and star added the following options:

  • -one-file if this option is in use in extract mode and a file matches, star exits after processing this file.

  • iseek=# is a byte offset to skip in the first read tape block before looking for an archive header.

  • mtseek=# This is a 512 byte based block offset that is based on the numbers printed with star -cv -block-offset .... If the archive is a plain file, star does a lseek() to that offset before starting to read. If the archive is a magnetic tape, star divides the block offset argument by the tape blocking factor and first issues an ioctl() that forwards the tape to that position, then implicitely computes the remainder as iseek offset in the first read block.

  • If you use star -xp mtseek=# -one-file pat=* ... star forwards the tape, reads the file and exits after restoring the file.

7
  • Yes, I managed to achieve this using the method you described but using dd with the larger tape block size, and then trimming the tar blocks off with another dd call and truncate call. Would that be a reasonable way to go about it? – swami Jun 4 '20 at 12:25
  • dd is expected to be slow compared to star, since star runs the tape I/O in a separate process that uses shared memory to the tar process and the os thus is not burdened with PIPE I/O. BTW: dou you really only like to extract one single file in this mode? This would make it fairly easy to implement. – schily Jun 4 '20 at 12:47
  • Ok, we have a fairly high spec server... So would have to test both. Yes this is our requirement. We are doing long term archiving of data on tape, so we like to archive in block based units...(each unit corresponds to a video shoot for one event). But often we need to retrieve individual clips for people, so this needs to be as efficient as possible, since the tape drives are heavily used – swami Jun 4 '20 at 13:09
  • So a new option-one-file (that causes star to exit after one file has been extracted via pattern or argument list) would be OK for you? – schily Jun 4 '20 at 13:26
  • Yes absolutely. – swami Jun 4 '20 at 13:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.