I have three kinds of data that are in different formats; for each data type, there is a Python script that transforms it into a single unified format.

This Python script is slow and CPU-bound (to a single core on a multi-core machine), so I want to run three instances of it - one for each data type - and combine their output to pass it into sort. Basically, equivalent to this:

{ ./handle_1.py; ./handle_2.py; ./handle_3.py } | sort -n

But with the three scripts running in parallel.

I found this question where GNU splitwas being used to round-robin some stdout stream between n instances of a script that handles the stream.

From the split man page:

-n, --number=CHUNKS
          generate CHUNKS output files.  See below
CHUNKS  may be:
 N       split into N files based on size of input
 K/N     output Kth of N to stdout
 l/N     split into N files without splitting lines
 l/K/N   output Kth of N to stdout without splitting lines
 r/N     like 'l'  but  use  round  robin  distributio

So the r/N command implies "without splitting lines".

Based on this, it seems like the following solution should be feasible:

split -n r/3 -u --filter="./choose_script" << EOF
> 1
> 2
> 3

Where choose_script does this:

{ read x; ./handle_$x.py; }

Unfortunately, I see some intermingling of lines - and lots of newlines that shouldn't be there.

For example, if I replace my Python scripts with some simple bash scripts that do this:

# ./handle_1.sh
while true; echo "1-$RANDOM"; done;


# ./handle_2.sh
while true; echo "2-$RANDOM"; done;


# ./handle_3.sh
while true; echo "3-$RANDOM"; done;

I see this output:



This is annoying - based on the man page extract I pasted above, it should maintain line integrity.

Obviously it works if I remove the -u argument, but then it's buffered and I'll run out of memory as it buffers the output of all but one of the scripts.

If anyone has some insight here it'd be greatly appreciated. I'm out of my depth here.

  • Some people in #bash on freenode suggested that I spawn all three processes and background them, writing to custom FDs, then loop over those FDs and read lines for them, but I haven't figured out how to make that workable. I was also told to look at the coproc builtin in bash, though I don't really see how it applies. – Cera Oct 25 '12 at 5:55
  • 1
    Do you have to do it without intermediate files? Couldn't you just do job1.py > file1 & job2.py > file 2 & job3.py > file3 ; wait ; sort -n file1 file2 file3 ? – angus Oct 25 '12 at 18:33

Try using the -u option of GNU parallel.

echo "1\n2\n3" | parallel -u -IX ./handle_X.sh

This runs them in parallel, without buffering the entirety of any process.

| improve this answer | |
  • I'm a little confused - is the X in IX telling -I that X will be the flag for replacing, or is it applying the -X flag, which seemingly also has a relevant meaning? – Cera Oct 25 '12 at 6:35
  • Hmph. I'm doing this: parallel -u -X ./handle_{}.sh ::: "1" "2" "3", and unfortunately I'm still seeing some output mangling. – Cera Oct 25 '12 at 6:43
  • the former: you can also use parallel -u ./handle_{}.sh, but I prefer to change it, since braces also have the meaning of joining together commands (as in your question). – flowblok Oct 25 '12 at 6:43
  • Seems to work for me, my grep doesn’t pick up any mangling: pastie.org/5113187 (are you using the test bash scripts, or your actual Python scripts?) – flowblok Oct 25 '12 at 6:49
  • The problem is that that's not actually doing anything in parallel. I'm using the bash scripts - pastie.org/5113225 – Cera Oct 25 '12 at 7:01


parallel ::: ./handle_1.py ./handle_2.py ./handle_3.py

If handle_1.py takes a file name:

parallel ::: ./handle_1.py ./handle_2.py ./handle_3.py ::: files*

You do not want the output mixed, so do not use -u.

If you want to keep the order (so all handle_1 output is before handle_2 and thus you might be able to avoid sorting):

parallel -k  ::: ./handle_1.py ./handle_2.py ./handle_3.py ::: files*

If you still want it sorted, you can parallelize the sort and utilize sort -m:

parallel --files "./handle_{1}.py {2} | sort -n"  ::: 1 2 3 ::: files* | parallel -j1 -X sort -m

Set $TMPDIR to a dir that is big enough to hold the output.

| improve this answer | |
  • 1
    I do want the output 'mixed' - I just want to make sure that every line in the final output is a single line from one of the subprocesses. If I dont mix it, the system will run out of memory buffering the stdout streams that aren't being printed out yet. – Cera Oct 25 '12 at 22:13
  • With GNU Parallel you will not run out of memory: It does not buffer in memory. Why do you think it buffers in memory? – Ole Tange Oct 27 '12 at 6:58

Maybe I'm missing something, but can't you just do:

(./handle_1.py & ./handle_2.py & ./handle_3.py) | sort -n

If you want lines from each process not to be interleaved, the easier is probably to make sure that the process themselves write them fully and possibly disable output buffering as writes to a pipe are guaranteed to be atomic as long as they're not bigger than PIPE_BUF. For instance, you could make sure it does use output buffering à la stdio and call fflush or whatever the equivalent is in python after one or a few lines have been written.

If you can't modify the python scripts, you could do:

lb() { grep --line-buffered '^'; }

(with GNU grep) or:

lb() while IFS= read -r l; do printf '%s\n' "$l"; done

(See notes in comments below if what the commands output is not text)

And do:

(./handle_1.py | lb & ./handle_2.py | lb & ./handle_3.py | lb) | sort -n

Another option to avoid those 3 lb processes is to have three pipes to one command that uses select/poll to see where there's some output coming from and feed it to sort line-based, but it takes a bit of programming.

| improve this answer | |
  • You need a wait in there, I think. – derobert Oct 25 '12 at 17:44
  • 1
    No unless some of the programs close their stdout before exiting, because the pipe and sort -n will remain until all programs that have an fd open on it have exited. – Stéphane Chazelas Oct 25 '12 at 17:51
  • Indeed, I tested, you are correct. – derobert Oct 25 '12 at 19:31
  • No, I still get mangled output. Lines get mixed together and interleaved. – Cera Oct 25 '12 at 22:16
  • 1
    OK @Cerales, see my updated answer – Stéphane Chazelas Oct 26 '12 at 6:13

Flowbok's answer was the correct solution. Oddly, the output of GNU parallel gets mangled if it's output directly to a file - but not if it goes to a tty.

Fortunately, script -c is available to mimic a tty.

There are still the three scripts:

# handle_1.sh
while true; do echo "1-$RANDOM$RANDOM$RANDOM$RANDOM"; done


# handle_2.sh
while true; do echo "2-$RANDOM$RANDOM$RANDOM$RANDOM"; done


# handle_3.sh
while true; do echo "3-$RANDOM$RANDOM$RANDOM$RANDOM"; done

Then there's a file that encapsulates the call to parallel:

# run_parallel.sh
parallel -u -I N ./handle_N.sh ::: "1" "2" "3"

And then I call it like this:

script -c ./run_parallel > output

The lines in output are mixed line-by-line between the output of the different scripts, but they don't get mangled or interleaved on a given line.

Bizarre behaviour from parallel - I may file a bug report.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.