8

I'd like to use the feature of GNU parallel where it can execute the command and the list it's fed in parallel and spit it out after it's all done, however, I don't want to install GNU parallel across all of our servers.

Or perhaps a parallel version of xargs?

Is there a ksh implementation of what GNU Parallel does? In this case, it doesn't have to be done in order as GNU Parallel does -- just as long as all the output can be piped or stored. I'd also like to avoid using temporary files.

  • GNU Parallel is a single perl script. I am curious why you prefer installing a ksh script over installing a single perl script that is designed to have very few dependencies. Can you elaborate on that? (Disclosure: I am the author of GNU Parallel) – Ole Tange Jun 21 '11 at 10:37
6

If you want to parallelize on a machine with multiple cores, you can just use (GNU) xargs, e.g.:

echo seq_[0-9][0-9].gz | xargs -n 1 -P 16 ./crunching

Meaning: xargs starts up to 16 processes in parallel of ./crunching using 1 token from stdin for each process.

You can also use split in combination with xargs.

Or you can create a simple Makefile for Job execution and call make -f mymf -j $CORES (you need temporary files for this solution).

PS: The GNU parallel manual also includes some comparisons with other tools, including xargs and make, interestingly they write:

(Very early versions of GNU parallel were coincidently implemented using make -j).

  • It is important for the asker that "all the output can be piped or stored". xargs is notoriously bad at that when run in parallel because the output can mix. – Ole Tange Jun 21 '11 at 10:40
  • @maxschlepzig: I agree with your answer 100%, I always just use xargs. Many people just don't know GNU xargs is able to do parallel! – J. M. Becker Dec 29 '11 at 19:09
0

Look at parallel --embed which embeds GNU Parallel into the same shell script that you use it from.

parallel --embed > new_script

then edit new_script.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.