GNU Parallel, without any command line options, allows you to easily parallelize a command whose last argument is determined by a line of STDIN:
$ seq 3 | parallel echo
2
1
3
Note that parallel
does not wait for EOF on STDIN before it begins executing jobs — running yes | parallel echo
will begin printing infinitely many copies of y
right away.
This behavior appears to change, however, if STDIN is relatively short:
$ { yes | ghead -n5; sleep 10; } | parallel echo
In this case, no output will be returned before sleep 10
completes.
This is just an illustration — in reality I'm attempting to read from a series of continually generated FIFO pipes where the FIFO-generating process will not continue until the existing pipes start to be consumed. For example, my command will produce a STDOUT stream like:
/var/folders/2b/1g_lwstd5770s29xrzt0bw1m0000gn/T/tmp.PFcggGR55i
/var/folders/2b/1g_lwstd5770s29xrzt0bw1m0000gn/T/tmp.UCpTBzI3J6
/var/folders/2b/1g_lwstd5770s29xrzt0bw1m0000gn/T/tmp.r2EmSLW0t9
/var/folders/2b/1g_lwstd5770s29xrzt0bw1m0000gn/T/tmp.5TRNeeZLmt
Manually cat
-ing each of these files one at a time in a new terminal causes the FIFO-generating process to complete successfully. However, running printfifos | parallel cat
does not work. Instead, parallel
seems to block forever waiting for input on STDIN — if I modify the pipeline to printfifos | head -n4 | parallel cat
, the deadlock disappears and the first four pipes are printed successfully.
This behavior seems to be connected to the --jobs|-j
parameter. Whereas { yes | ghead -n5; sleep 10; } | parallel cat
produces no output for 10 seconds, adding a -j1
option yields four lines of y
almost immediately followed by a 10 second wait for the final y
. Unfortunately this does not solve my problem — I need every argument to be processed before parallel
can get EOF from reading STDIN. Is there any way to achieve this?
unbuffer printfifos
make a difference?stdbuf printfifos
?unbuffer bash -c 'unbuffer seq 3; unbuffer sleep 10' | unbuffer -p parallel -j1 echo
also exhibits the problem as well.parallel
, and its source code is quite a maze, but one thing seems certain:parallel
isn't select(2)ing on the fds it's reading the arguments from (and it isn't using any async i/o either), but it's always doing a blocking read (look at$arg = <$fh>
inread_arg_from_fh
in its source code). So once it got there one way or another, it will have to wait until either more data or EOF is coming. I hope I'm wrong, but it looks like you have to re-evaluate your approach ;-)