Archiving stdout to multiple tapes

Question

I have large files which are generated on the fly to stdout, one every 24hours. I would like to archive these files progressively on tapes, ideally in a single archive which potentially spans multiple tapes.

Tar is very good for managing the tapes, as it has built-in functionalities to append to an archive and to load the next tape. But it is very poor at accepting data from stdin. No matter what I do, it ends up writing a special file (link or named pipe) to the archive, instead of its content.

Here is the example command, that I have been trying. The first day, generate a new archive:

ln -s /dev/stdin day1 # or use the --transform option of tar
data_generator | tar -c -h -M -f /dev/nst0 -H posix -F 'mtx -f /dev/sch0 next' day1

the next day, I would like to just change -c to -A and save the new stream into a new file appended to the tar archive, loading a new tape when it becomes necessary.

data_generator | tar -A -h -M -f /dev/nst0 -H posix -F 'mtx -f /dev/sch0 next' day2

As I said, all I find in the archive is a named pipe (with -h) or a symlink (without -h).

Some ideas that I have tried and are not good:

1. Using split instead of tar is not viable, because it is too basic. It can only split to pre-defined dimension (not good if I do not start from the beginning of the tape), and it cannot concatenate the different days in an unpackable archive. Tar does not need to know the size of the data nor the tape, it will just switch to a new tape when it gets a write error.
2. I've read the manuals of cpio, star and dar. I do not get the impression that they cope with pipes better than tar.

Thank you for any hints.

Edit: I'm starting to think that it is impossible with tar, because it needs to know the size of the file before starting to write. In fact, an archive that can be expanded, appending is very tricky if you do write the size before the content.

Answer 1

After one night of rest, I found that this can be done with a bit of python if one knows in advance the amount of data to be read (I do). This simple program reads 1MB of data from stdin, and write to stdout a tar archive encapsulating the data as "filename.dat".

#!/usr/bin/env python3
import tarfile
with open("/dev/stdout", "ab") as outf:
  tar = tarfile.open(fileobj=outf, mode="w")
  with open("/dev/stdin", "rb") as inf:
    filesize=1048576 # 1MB
    tarinfo = tarfile.TarInfo(name="filename.dat")
    tarinfo.size=filesize
    tar.addfile(tarinfo, fileobj=inf)
# -- end program tarpipe.py ---

For example, you can get 1MB of random data and pipe it though this program to a tar archive:

cat /dev/urandom |./tarpipe.py > daily.tar

The resulting archive contains a single file of 1MB named "filename.dat".

Because of how tar archives works, you can just keep appending (>>) to extend it, as long as the filename is modified each time (or the files will overwrite each other when extracting).

In order to manage tape changing, I can pipe the output through mbuffer which can do it natively.