I have an embarrassingly parallel process that creates a huge amount of nearly (but not completely) identical files. Is there a way to archive the files "on the fly", so that the data does not consume more space than necessary?

The process itself accepts command-line parameters and prints the name of each file created to stdout. I'm invoking it with parallel --gnu which takes care of distributing input (which comes from another process) and collecting output:

arg_generating_process | parallel --gnu my_process | magic_otf_compressor

SIMPLE EXAMPLE for the first part of the pipe in bash:

for ((f = 0; $f < 100000; f++)); do touch $f; echo $f; done

How could magic_otf_compressor look like? It's supposed to treat each input line as file name, copy each file to a compressed .tar archive (the same archive for all files processed!) and then delete it. (Actually, it should be enough to print the name of each processed file, another | parallel --gnu rm could take care of deleting the files.)

Is there any such tool? I'm not considering compressing each file individually, this would waste far too much space. I have looked into archivemount (will keep file system in memory -> impossible, my files are too large and too many) and avfs (couldn't get it to work together with FUSE). What have I missed?

I'm just one step away from hacking such a tool myself, but somebody must have done it before...

EDIT: Essentially I think I'm looking for a stdin front-end for libtar (as opposed to the command-line front-end tar that reads arguments from, well, the command line).

  • Have you considered writing files in a format that has native compression? E.g. hdf5 can be compressed as they are written with either gzip or szip compression. Hdf5 also supports MPI so it works well with those embarrassingly parallel problems.
    – casey
    Mar 11, 2014 at 17:18
  • 2
    If you want compression and deduplication, zfs comes to mind. Mar 11, 2014 at 17:27
  • @casey: It's HTML, but I suppose I could use an HDF5 container.? Haven't considered this yet.
    – krlmlr
    Mar 11, 2014 at 17:54
  • @StephaneChazelas: Can this be implemented in userland?
    – krlmlr
    Mar 11, 2014 at 17:54

4 Answers 4


A classic case of RTFM (all of it!). The -T option to GNU tar will read the files to be archived from another file (in my case, /dev/stdin, you can also use -), and there's even a --remove-files option:

alias magic_otf_compressor='tar --create -T - --remove-files -O | pixz'

(using the parallel version of xz for compression, but you can use your preferred compressor instead). To be used as:

arg_generating_process |
  parallel --gnu my_process |
  magic_otf_compressor > file.tar.xz

EDIT: As Ole points out, tar seems to read the entire list of files with the -T option for some reason. The following test confirms this:

for ((f = 0; $f < 1000; f++)); do
    touch $f; echo $f;
done | tar -c -f otf.tar -T - -v

There is a one second delay on my system before all files are printed at once; in contrast, if the tar command is replaced by cat, all files are printed as they are created. I have filed a support request with the tar folks, let's see.

EDIT^2: The most recent tar from source fixes this. It's not in Ubuntu 13.10 yet, but might be included with 14.04.


It seems tar wants to know all the file names upfront. So it is less on-the-fly and more after-the-fly. cpio does not seem to have that problem:

| cpio -vo 2>&1 > >(gzip > /tmp/arc.cpio.gz) | parallel rm
  • Thanks. So, even RTFM is not enough ;-) I even looked into tar's code to see that there's a function that returns the next file name to be processed, which made me read the documentation again. -- So, stdout is directed to the gzip process via process substitution, and stderr is redirected to stdout which is processed by the next step in the pipe?
    – krlmlr
    Mar 12, 2014 at 8:39
  • Yup. The > >() construct does not work in all shells, but it works in Bash.
    – Ole Tange
    Mar 12, 2014 at 9:17
  • I can confirm that tar reads the file list first, using the simple example I've added to my question. However, reading tar's source code again, it seems to me that it should read the list of files "on the fly" if not creating an incremental archive. Unfortunately, I have errors compiling tar from source... :-(
    – krlmlr
    Mar 12, 2014 at 9:20
  • I haven't found a way to suppress the final line in the output of cpio, other than grep -v 'blocks$'. (head -n -1 uses a very large buffer...) Makes this solution a bit of a hack, but never mind ;-)
    – krlmlr
    Mar 12, 2014 at 15:40
  • @krlmlr that is odd: My head -n -1 only uses 16MB when run on a few GB of data. You can always use perl: perl -ne 'print $last;$last=$_'
    – Ole Tange
    Mar 14, 2014 at 9:46

Somehow this doesn't seem a good job for a solid compressor (tape-based archivers + compression). Inserting files one after another looks like a job for zip or some other format that allows random file access within the archive and incremental insertion.

The fact that the files are similar won't help much in either case. In zip, files are compressed separately, and in solid compressors, there is usually a window within which the compression takes place.

If the files are text-based, you could store diffs compared to a single reference file. For binary, It's a bit more tricky but can be done.

There's also a formal way (not write-only, but proper filesystems). For instance, ZFS and BTRFS filesystems offer transparent compression. You could also use this http://developer.berlios.de/projects/fusecompress

  • My files are about 100k each. Wouldn't be enough to allow the compressor to use a window of, say, 1M? xz seems to operate with a default dictionary size of 8M (at default compression level -6), which seems to be plenty for my use case. -- Diffs to a reference file are nice, but requires to construct a reference file first. Would a compressing file system detect files with near-identical content?
    – krlmlr
    Mar 12, 2014 at 10:35
  • Compressing file systems don't compress across files (nor does zip), but btrfs does have copy-on-write, so if you copy a file and modify a part of it, it only saves parts that you changed. If you are not creating files this way, there supposedly exist deduplication tools, but btrfs is not a mature and stable filesystem yet and deduplication this is in early development stages. But now I think of it, what about lessfs.com/wordpress
    – orion
    Mar 12, 2014 at 10:41
  • I do get impressive compression ratios with a solid compressor for my use case, but, as you outlined, I assume the results would be worse if the files were larger than the dictionary size.
    – krlmlr
    Mar 14, 2014 at 7:48

It may not seem obvious, but I bet squashfs would be perfect for this - and it's even implemented in kernel. Since version 4.1 squashfs can handle pseudo files as specified on the mksquash command line or via a shell-script and mksquashfs will generate the files as it creates the archive.

It can handle pipes - for instance, you can capture another process's stdout into a mountable squash archive - even fifos - it's pretty cool. In your case, if you could work out the script logistics of piping your process's output through it, you could wrap your process entirely in mksquashfs and wind up with a single archive. Here's a bit from the readme on how it works and there's more here:

Mksquashfs 4.1 adds support for "dynamic pseudo files" and a modify operation. Dynamic pseudo files allow files to be dynamically created when Mksquashfs is run, their contents being the result of running a command or piece of shell script. The modifiy operation allows the mode/uid/gid of an existing file in the source filesystem to be modified.

Creating dynamic file examples

Create a file "dmesg" containing the output from dmesg.

    dmesg f 444 root root dmesg

Create a file RELEASE containing the release name, date, build host, and an incrementing version number. The incrementing version is a side-effect of executing the shell script, and ensures every time Mksquashfs is run a new version number is used without requiring any other shell scripting.

    RELEASE f 444 root root \
        if [ ! -e /tmp/ver ]; then \
        echo 0 > /tmp/ver; \
        fi; \
        ver=`cat /tmp/ver`; \
            ver=$((ver +1)); \
            echo $ver > /tmp/ver; \
            echo -n "release x.x"; \
            echo "-dev #"$ver `date` "Build host" `hostname`

Copy 10K from the device /dev/sda1 into the file input. Ordinarily Mksquashfs given a device, fifo, or named socket will place that special file within the Squashfs filesystem, this allows input from these special files to be captured and placed in the Squashfs filesystem.

        input f 444 root root dd if=/dev/sda1 bs=1024 count=10
  • How would this work within the infrastructure I outlined?
    – krlmlr
    Mar 12, 2014 at 13:06
  • You would have to get your process to write its filenames to mksquash's invocation script, and have it continue to append them as it runs. Or even into a tmpfs that squash will read and compress as it runs. Or, as another mentioned, through something else - invoke cpio just like the above dd example, but with cpio use its copy function maybe. In any case - it definitely reads, creates, and compresses on the fly.
    – mikeserv
    Mar 12, 2014 at 13:09
  • Will it compress across files?
    – krlmlr
    Mar 12, 2014 at 13:14
  • It compresses its input in a stream - all inodes, all of it. I've used it with dd and it was pretty cool - I always use the 1MB block-size and xz compression.
    – mikeserv
    Mar 12, 2014 at 13:15
  • This looks like an option, but from your answer I fail to see how to create, say, a squashfs archive with a directory test and a file file in this directory. Could you please provide a brief example?
    – krlmlr
    Mar 14, 2014 at 7:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.