4

GNU Parallel, without any command line options, allows you to easily parallelize a command whose last argument is determined by a line of STDIN:

$ seq 3 | parallel echo
2
1
3

Note that parallel does not wait for EOF on STDIN before it begins executing jobs — running yes | parallel echo will begin printing infinitely many copies of y right away.

This behavior appears to change, however, if STDIN is relatively short:

$ { yes | ghead -n5; sleep 10; } | parallel echo

In this case, no output will be returned before sleep 10 completes.

This is just an illustration — in reality I'm attempting to read from a series of continually generated FIFO pipes where the FIFO-generating process will not continue until the existing pipes start to be consumed. For example, my command will produce a STDOUT stream like:

/var/folders/2b/1g_lwstd5770s29xrzt0bw1m0000gn/T/tmp.PFcggGR55i
/var/folders/2b/1g_lwstd5770s29xrzt0bw1m0000gn/T/tmp.UCpTBzI3J6
/var/folders/2b/1g_lwstd5770s29xrzt0bw1m0000gn/T/tmp.r2EmSLW0t9
/var/folders/2b/1g_lwstd5770s29xrzt0bw1m0000gn/T/tmp.5TRNeeZLmt

Manually cat-ing each of these files one at a time in a new terminal causes the FIFO-generating process to complete successfully. However, running printfifos | parallel cat does not work. Instead, parallel seems to block forever waiting for input on STDIN — if I modify the pipeline to printfifos | head -n4 | parallel cat, the deadlock disappears and the first four pipes are printed successfully.

This behavior seems to be connected to the --jobs|-j parameter. Whereas { yes | ghead -n5; sleep 10; } | parallel cat produces no output for 10 seconds, adding a -j1 option yields four lines of y almost immediately followed by a 10 second wait for the final y. Unfortunately this does not solve my problem — I need every argument to be processed before parallel can get EOF from reading STDIN. Is there any way to achieve this?

7
  • Does unbuffer printfifos make a difference? stdbuf printfifos? Commented Apr 7, 2019 at 8:04
  • No effect — unbuffer bash -c 'unbuffer seq 3; unbuffer sleep 10' | unbuffer -p parallel -j1 echo also exhibits the problem as well.
    – goodside
    Commented Apr 7, 2019 at 8:20
  • I'm not familiar with parallel, and its source code is quite a maze, but one thing seems certain: parallel isn't select(2)ing on the fds it's reading the arguments from (and it isn't using any async i/o either), but it's always doing a blocking read (look at $arg = <$fh> in read_arg_from_fh in its source code). So once it got there one way or another, it will have to wait until either more data or EOF is coming. I hope I'm wrong, but it looks like you have to re-evaluate your approach ;-)
    – user313992
    Commented Apr 7, 2019 at 14:56
  • And btw, you cannot "send" an EOF. The only way to cause an EOF on the reading end of a pipe is to close all the open handles to its writing end.
    – user313992
    Commented Apr 7, 2019 at 14:58
  • (I know EOF isn't a valid byte, but in the same sense you say "either more data or EOF is coming" it's ubiquitous to describe EOF as metaphorically moving through a pipeline, especially when there are no concurrent writers. I think it's clear what I meant, but I'll edit to avoid confusing beginners.)
    – goodside
    Commented Apr 7, 2019 at 15:59

1 Answer 1

2

A bug in GNU Parallel does, that it only starts processing after having read one job for each jobslot. After that it reads one job at a time.

In older versions the output will also be delayed by the number of jobslots. Newer versions only delay output by a single job.

So if you sent one job per second to parallel -j10 it would read 10 jobs before starting them. Older versions you would then have to wait an additional 10 seconds before seeing the output from job 3.

A workaround the limitation at start is to feed one dummy job per jobslot to parallel:

true >jobqueue; tail -n+0 -f jobqueue | parallel &
seq $(parallel --number-of-threads) | parallel -N0 echo true >> jobqueue
# now add the real jobs to jobqueue

A workound the output is to use --linebuffer (but this will mix full lines from different jobs).

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .