Using GNU parallel on line based output without temporary files

Question

The default output mode of GNU parallel is --group:
The output of each job is written to a temporary file and passed to the output of parallel only after the job has finished.

When using this default output mode on data larger than the /tmp space like in
parallel lz4 -dc ::: /var/lib/apt/lists/*lz4 | wc it is slow and crashes with
parallel: Error: Output is incomplete.
Cannot append to buffer file in /tmp.

When using the --ungroup mode, lines are split in the middle which causes a different output from
parallel --ungroup lz4 -dc ::: /var/lib/apt/lists/*lz4 | wc
than from unparallelized lz4 -dmc /var/lib/apt/lists/*lz4 | wc.

According to the parallel manpage, this should be solved by the --line-buffer option as I understand it: All jobs have an output pipe which is read by parallel and if output becomes available from any job, it gets passed line by line to the output pipe of the parallel process itself. (Edit: I mean lines in blocks as is done for spreading large input to the parallel processes, not one syscall per line which would be far too slow.)

But this doesn't work:
parallel --line-buffer lz4 -dc ::: /var/lib/apt/lists/*lz4 | wc -c
results in the same disk full error as with the implied --group above.

How to use parallel --line-buffer without temporary files ?

System is LUbuntu 20 LTS. parallel -V returns 20161222. Comparision of raw serial and parallel decompression performance on Dual-Core i3-4130 with hyper threading (4 threads):

time ls -S /var/lib/apt/lists/*lz4 | parallel --ungroup lz4 -dc > /dev/null
1.461s
time lz4 -dmc /var/lib/apt/lists/*lz4 > /dev/null
3.069s

The real use case is as follows (workarond without --line-buffer):

time lz4 -dmc /var/lib/apt/lists/*Contents* | grep -F $'/parallel\t' | sort -u
usr/bin/parallel                                            universe/utils/moreutils,universe/utils/parallel
usr/bin/parallel                                            universe/utils/parallel
usr/lib/R/library/parallel/R/parallel                       universe/math/r-base-core
usr/lib/cups/backend/parallel                               net/cups-filters
usr/share/doc-base/parallel                                 universe/utils/parallel
real    0m5.349s
user    0m3.970s
sys     0m5.839s

time ls -S /var/lib/apt/lists/*Contents* | parallel lz4 -dc '{}' \| grep -F "\$'/parallel\t'" | sort -u
(same output as above)
real    0m3.669s
user    0m5.888s
sys     0m7.676s

This parallelizes not only the decompression but also the postprocessing and is the better solution here where the work is not 99 % in the first part of the pipe.
But this approach to parallelize the complete pipe is not always possible, so the general question remains open for cases where output of first step is not very small and thus streaming is wanted.

Answer 1

To do what you suggest, it would need the output of each lz4 to go to separate pipes, and either a select/poll loop that reads from all and splits their output on lines or have one process / thread handling each pipe.

That sounds like significant overhead. Even without that overhead I find that on my 12 year old 4 core 8 thread laptop with a fast SSD, printf '%s\0' /var/lib/apt/lists/*lz4 | xargs -r0 -n 1 -P8 lz4 -dc (even without the GNU parallel overhead) is not faster than lz4 -dmc /var/lib/apt/lists/*lz4.

Ideally here, you'd want for the parallelised command to output line-buffered in the first place. With many, you can do it with stdbuf -oL.

That doesn't seem to be the case of lz4 though, but you could implement the second approach by hand (one process handling one lz4 output) with:

printf '%s\0' /var/lib/apt/lists/*lz4 |
  stdbuf -oL xargs -r0 -n 1 -P4 sh -c 'lz4 -dc "$1" | paste' sh | 
  wc -c

(paste being a command that processes its input one line at a time and can be convinced to do line-buffering on output with stdbuf, see also GNU grep's grep --line-buffered '^'; avoid sed -u which outputs one line at a time but also reads its input one byte at a time).

Even with the output discarded to /dev/null, on my system, that's 13 times as slow as the non-parallelised lz4 -dmc /var/lib/apt/lists/*lz4 (6.5s vs 0.5s).

And that's using paste that's written in C. GNU parallel being written in perl, if does support something like that internally, there's good chance it will be even less efficient.

Parallelising (at least in that fashion) would only make sense for CPU intensive tasks that produce relatively little text output which is the opposite of lz4 which is meant to decompress with little effort.

Answer 2

The release notes for version 20170822 says:

• --line-buffer no longer uses tempfiles. This is faster and makes it possible for a single process to output more data than there is free disk space.

So the solution is to upgrade to 20170822.

GNU Parallel still uses temporary files for other stuff, but not for the line buffering.

It will also do The Right Thing if you have long lines:

#!/bin/bash

5gfile() {
    # Create file with 5GB long line
    perl -e '$a=(shift)x1000000;for(1..5000){print $a};print "\n"' $1 | lz4 > $1.lz4;
}
export -f 5gfile
parallel 5gfile ::: a b c d

echo The correct output: One line with a b c d
lz4 -dc {a..d}.lz4 | tr -s abcd

echo Output from parallel: One line with a b c d might be reordered
parallel --line-buffer lz4 -dc ::: {a..d}.lz4 |
    tr -s abcd

echo Output from xargs with stdbuf -oL
echo This does not handle long lines because stdbuf -oL does not guarantee only full lines will be written
printf '%s\0' /tmp/*lz4 |
    stdbuf -oL xargs -r0 -n 1 -P4 sh -c 'lz4 -dc "$1" | paste' sh |
    tr -s abcd