10

I have three kinds of data that are in different formats; for each data type, there is a Python script that transforms it into a single unified format.

This Python script is slow and CPU-bound (to a single core on a multi-core machine), so I want to run three instances of it - one for each data type - and combine their output to pass it into sort. Basically, equivalent to this:

{ ./handle_1.py; ./handle_2.py; ./handle_3.py } | sort -n

But with the three scripts running in parallel.

I found this question where GNU splitwas being used to round-robin some stdout stream between n instances of a script that handles the stream.

From the split man page:

-n, --number=CHUNKS
          generate CHUNKS output files.  See below
CHUNKS  may be:
 N       split into N files based on size of input
 K/N     output Kth of N to stdout
 l/N     split into N files without splitting lines
 l/K/N   output Kth of N to stdout without splitting lines
 r/N     like 'l'  but  use  round  robin  distributio

So the r/N command implies "without splitting lines".

Based on this, it seems like the following solution should be feasible:

split -n r/3 -u --filter="./choose_script" << EOF
> 1
> 2
> 3
> EOF

Where choose_script does this:

#!/bin/bash
{ read x; ./handle_$x.py; }

Unfortunately, I see some intermingling of lines - and lots of newlines that shouldn't be there.

For example, if I replace my Python scripts with some simple bash scripts that do this:

#!/bin/bash
# ./handle_1.sh
while true; echo "1-$RANDOM"; done;

.

#!/bin/bash
# ./handle_2.sh
while true; echo "2-$RANDOM"; done;

.

#!/bin/bash
# ./handle_3.sh
while true; echo "3-$RANDOM"; done;

I see this output:

1-8394

2-11238
2-22757
1-723
2-6669
3-3690
2-892
2-312511-24152
2-9317
3-5981

This is annoying - based on the man page extract I pasted above, it should maintain line integrity.

Obviously it works if I remove the -u argument, but then it's buffered and I'll run out of memory as it buffers the output of all but one of the scripts.

If anyone has some insight here it'd be greatly appreciated. I'm out of my depth here.

2
  • Some people in #bash on freenode suggested that I spawn all three processes and background them, writing to custom FDs, then loop over those FDs and read lines for them, but I haven't figured out how to make that workable. I was also told to look at the coproc builtin in bash, though I don't really see how it applies.
    – Cera
    Oct 25, 2012 at 5:55
  • 1
    Do you have to do it without intermediate files? Couldn't you just do job1.py > file1 & job2.py > file 2 & job3.py > file3 ; wait ; sort -n file1 file2 file3 ?
    – angus
    Oct 25, 2012 at 18:33

4 Answers 4

2

Try using the -u option of GNU parallel.

echo "1\n2\n3" | parallel -u -IX ./handle_X.sh

This runs them in parallel, without buffering the entirety of any process.

8
  • I'm a little confused - is the X in IX telling -I that X will be the flag for replacing, or is it applying the -X flag, which seemingly also has a relevant meaning?
    – Cera
    Oct 25, 2012 at 6:35
  • Hmph. I'm doing this: parallel -u -X ./handle_{}.sh ::: "1" "2" "3", and unfortunately I'm still seeing some output mangling.
    – Cera
    Oct 25, 2012 at 6:43
  • the former: you can also use parallel -u ./handle_{}.sh, but I prefer to change it, since braces also have the meaning of joining together commands (as in your question).
    – flowblok
    Oct 25, 2012 at 6:43
  • Seems to work for me, my grep doesn’t pick up any mangling: pastie.org/5113187 (are you using the test bash scripts, or your actual Python scripts?)
    – flowblok
    Oct 25, 2012 at 6:49
  • The problem is that that's not actually doing anything in parallel. I'm using the bash scripts - pastie.org/5113225
    – Cera
    Oct 25, 2012 at 7:01
2

Try:

parallel ::: ./handle_1.py ./handle_2.py ./handle_3.py

If handle_1.py takes a file name:

parallel ::: ./handle_1.py ./handle_2.py ./handle_3.py ::: files*

You do not want the output mixed, so do not use -u.

If you want to keep the order (so all handle_1 output is before handle_2 and thus you might be able to avoid sorting):

parallel -k  ::: ./handle_1.py ./handle_2.py ./handle_3.py ::: files*

If you still want it sorted, you can parallelize the sort and utilize sort -m:

parallel --files "./handle_{1}.py {2} | sort -n"  ::: 1 2 3 ::: files* | parallel -j1 -X sort -m

Set $TMPDIR to a dir that is big enough to hold the output.

2
  • 1
    I do want the output 'mixed' - I just want to make sure that every line in the final output is a single line from one of the subprocesses. If I dont mix it, the system will run out of memory buffering the stdout streams that aren't being printed out yet.
    – Cera
    Oct 25, 2012 at 22:13
  • With GNU Parallel you will not run out of memory: It does not buffer in memory. Why do you think it buffers in memory?
    – Ole Tange
    Oct 27, 2012 at 6:58
2

Maybe I'm missing something, but can't you just do:

(./handle_1.py & ./handle_2.py & ./handle_3.py) | sort -n

If you want lines from each process not to be interleaved, the easier is probably to make sure that the process themselves write them fully and possibly disable output buffering as writes to a pipe are guaranteed to be atomic as long as they're not bigger than PIPE_BUF. For instance, you could make sure it does use output buffering à la stdio and call fflush or whatever the equivalent is in python after one or a few lines have been written.

If you can't modify the python scripts, you could do:

lb() { grep --line-buffered '^'; }

(with GNU grep) or:

lb() while IFS= read -r l; do printf '%s\n' "$l"; done

(See notes in comments below if what the commands output is not text)

And do:

(./handle_1.py | lb & ./handle_2.py | lb & ./handle_3.py | lb) | sort -n

Another option to avoid those 3 lb processes is to have three pipes to one command that uses select/poll to see where there's some output coming from and feed it to sort line-based, but it takes a bit of programming.

7
  • You need a wait in there, I think.
    – derobert
    Oct 25, 2012 at 17:44
  • 1
    No unless some of the programs close their stdout before exiting, because the pipe and sort -n will remain until all programs that have an fd open on it have exited. Oct 25, 2012 at 17:51
  • Indeed, I tested, you are correct.
    – derobert
    Oct 25, 2012 at 19:31
  • No, I still get mangled output. Lines get mixed together and interleaved.
    – Cera
    Oct 25, 2012 at 22:16
  • 1
    OK @Cerales, see my updated answer Oct 26, 2012 at 6:13
1

Flowbok's answer was the correct solution. Oddly, the output of GNU parallel gets mangled if it's output directly to a file - but not if it goes to a tty.

Fortunately, script -c is available to mimic a tty.

There are still the three scripts:

#!/bin/bash
# handle_1.sh
while true; do echo "1-$RANDOM$RANDOM$RANDOM$RANDOM"; done

.

#!/bin/bash
# handle_2.sh
while true; do echo "2-$RANDOM$RANDOM$RANDOM$RANDOM"; done

.

#!/bin/bash
# handle_3.sh
while true; do echo "3-$RANDOM$RANDOM$RANDOM$RANDOM"; done

Then there's a file that encapsulates the call to parallel:

#!/bin/bash
# run_parallel.sh
parallel -u -I N ./handle_N.sh ::: "1" "2" "3"

And then I call it like this:

script -c ./run_parallel > output

The lines in output are mixed line-by-line between the output of the different scripts, but they don't get mangled or interleaved on a given line.

Bizarre behaviour from parallel - I may file a bug report.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .