Virtual write-only file system for storing files in archive

Question

I have an embarrassingly parallel process that creates a huge amount of nearly (but not completely) identical files. Is there a way to archive the files "on the fly", so that the data does not consume more space than necessary?

The process itself accepts command-line parameters and prints the name of each file created to stdout. I'm invoking it with parallel --gnu which takes care of distributing input (which comes from another process) and collecting output:

arg_generating_process | parallel --gnu my_process | magic_otf_compressor

SIMPLE EXAMPLE for the first part of the pipe in bash:

for ((f = 0; $f < 100000; f++)); do touch $f; echo $f; done

How could magic_otf_compressor look like? It's supposed to treat each input line as file name, copy each file to a compressed .tar archive (the same archive for all files processed!) and then delete it. (Actually, it should be enough to print the name of each processed file, another | parallel --gnu rm could take care of deleting the files.)

Is there any such tool? I'm not considering compressing each file individually, this would waste far too much space. I have looked into archivemount (will keep file system in memory -> impossible, my files are too large and too many) and avfs (couldn't get it to work together with FUSE). What have I missed?

I'm just one step away from hacking such a tool myself, but somebody must have done it before...

EDIT: Essentially I think I'm looking for a stdin front-end for libtar (as opposed to the command-line front-end tar that reads arguments from, well, the command line).

Answer 1

A classic case of RTFM (all of it!). The -T option to GNU tar will read the files to be archived from another file (in my case, /dev/stdin, you can also use -), and there's even a --remove-files option:

alias magic_otf_compressor='tar --create -T - --remove-files -O | pixz'

(using the parallel version of xz for compression, but you can use your preferred compressor instead). To be used as:

arg_generating_process |
  parallel --gnu my_process |
  magic_otf_compressor > file.tar.xz

EDIT: As Ole points out, tar seems to read the entire list of files with the -T option for some reason. The following test confirms this:

for ((f = 0; $f < 1000; f++)); do
    touch $f; echo $f;
done | tar -c -f otf.tar -T - -v

There is a one second delay on my system before all files are printed at once; in contrast, if the tar command is replaced by cat, all files are printed as they are created. I have filed a support request with the tar folks, let's see.

EDIT^2: The most recent tar from source fixes this. It's not in Ubuntu 13.10 yet, but might be included with 14.04.

Answer 2

It seems tar wants to know all the file names upfront. So it is less on-the-fly and more after-the-fly. cpio does not seem to have that problem:

| cpio -vo 2>&1 > >(gzip > /tmp/arc.cpio.gz) | parallel rm

Answer 3

Somehow this doesn't seem a good job for a solid compressor (tape-based archivers + compression). Inserting files one after another looks like a job for zip or some other format that allows random file access within the archive and incremental insertion.

The fact that the files are similar won't help much in either case. In zip, files are compressed separately, and in solid compressors, there is usually a window within which the compression takes place.

If the files are text-based, you could store diffs compared to a single reference file. For binary, It's a bit more tricky but can be done.

There's also a formal way (not write-only, but proper filesystems). For instance, ZFS and BTRFS filesystems offer transparent compression. You could also use this http://developer.berlios.de/projects/fusecompress

Answer 4

It may not seem obvious, but I bet squashfs would be perfect for this - and it's even implemented in kernel. Since version 4.1 squashfs can handle pseudo files as specified on the mksquash command line or via a shell-script and mksquashfs will generate the files as it creates the archive.

It can handle pipes - for instance, you can capture another process's stdout into a mountable squash archive - even fifos - it's pretty cool. In your case, if you could work out the script logistics of piping your process's output through it, you could wrap your process entirely in mksquashfs and wind up with a single archive. Here's a bit from the readme on how it works and there's more here:

Mksquashfs 4.1 adds support for "dynamic pseudo files" and a modify operation. Dynamic pseudo files allow files to be dynamically created when Mksquashfs is run, their contents being the result of running a command or piece of shell script. The modifiy operation allows the mode/uid/gid of an existing file in the source filesystem to be modified.

Creating dynamic file examples

Create a file "dmesg" containing the output from dmesg.

    dmesg f 444 root root dmesg

Create a file RELEASE containing the release name, date, build host, and an incrementing version number. The incrementing version is a side-effect of executing the shell script, and ensures every time Mksquashfs is run a new version number is used without requiring any other shell scripting.

    RELEASE f 444 root root \
        if [ ! -e /tmp/ver ]; then \
        echo 0 > /tmp/ver; \
        fi; \
        ver=`cat /tmp/ver`; \
            ver=$((ver +1)); \
            echo $ver > /tmp/ver; \
            echo -n "release x.x"; \
            echo "-dev #"$ver `date` "Build host" `hostname`

Copy 10K from the device /dev/sda1 into the file input. Ordinarily Mksquashfs given a device, fifo, or named socket will place that special file within the Squashfs filesystem, this allows input from these special files to be captured and placed in the Squashfs filesystem.

        input f 444 root root dd if=/dev/sda1 bs=1024 count=10