Don't get depressed over this... I'm sure you'll manage to solve this problem!
I think what's tripping you with tar is that by "supports named pipes" they mean that it can recognize the named pipes and store them as named pipes inside the archive, so that you can later restore them as named pipes again... Which is not really what you want.
Also, the format of tar files is not very suitable for what you are doing, since the entry describing a file is stored before its contents and the file entry must contain the file size, so unless you know the exact size of the contents in advance, it's hard to do that...
There is this solution (see TarFileStdin), which uses a hack to solve that problem. It inserts a TarInfo with file size zero, then stores the contents of the file and finally seeks back to the offset of the original TarInfo and overwrites it with the correct size... It's a bit too hacky, but it should work... But read on.
You mentioned "I want to be able to index the tarball, so that I can extract the metadata file quickly from the .tar.xz file", so that looks more like a ZIP file! The ZIP format stores the contents of all files first and then stores a table of file information and offsets at the end of the ZIP. In that sense, it is indexed, like you mentioned. Listing the contents of the ZIP can be done quickly, since it's easy for tools to find that file table by starting from the end of the file.
You could use ZIP's native compression format, or you could just use ZIP's "store" mode (uncompressed) and then add a xyz.pcap.xz file inside it. Adding a *.xz file to the ZIP would have the convenience that you can use an external compressor, such as a parallel xz.
zipfile.ZipFile objects have an
open() method which allows you to add a file by name only and receive a file object to which you can write the contents.
You could use that API and
shutil.copyfileobj() to add your compressed pcap from a named pipe to the ZIP file:
with zipfile.ZipFile('mydata.zip', 'w') as zf:
with zf.open('xyz.pcap.xz', 'w') as outputf:
with open('/path/to/namedpipe', 'r') as inputf:
zf.write('metadata.pickle') # from local directory
This code snippet is assuming you're writing the already xz-compressed pcap data to the named pipe and that you have the metadata already serialized to a file named 'metadata.pickle' in the current directory. (Of course, you could use ZipFile's
open() to serialize the pickle metadata directly into an entry in the ZIP file too!)
If you want to use zipfile's native compression, you can set a default compression to ZipFile:
with zipfile.ZipFile('mydata.zip', 'w', zipfile.ZIP_LZMA) as zf:
(The default is ZIP_STORED, which means no compression, which is probably what you want if you're gonna pipe xz-compressed data there.)
See the documentation of zipfile for more details. Newer Python's have even more features, for instance with Python 3.5 you can actually write the zipfile to a pipe so you could for instance upload it straight to a remote host through SSH.
I hope you find this useful! If you really need a tarball, try this answer, but I really think the zipfile solution using Python 3 is a better approach for the use case you describe! So if that format is a possibility, I'd really recommend it.