Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I'd like to use the feature of GNU parallel where it can execute the command and the list it's fed in parallel and spit it out after it's all done, however, I don't want to install GNU parallel across all of our servers.

Or perhaps a parallel version of xargs?

Is there a ksh implementation of what GNU Parallel does? In this case, it doesn't have to be done in order as GNU Parallel does -- just as long as all the output can be piped or stored. I'd also like to avoid using temporary files.

share|improve this question
    
GNU Parallel is a single perl script. I am curious why you prefer installing a ksh script over installing a single perl script that is designed to have very few dependencies. Can you elaborate on that? (Disclosure: I am the author of GNU Parallel) –  Ole Tange Jun 21 '11 at 10:37

1 Answer 1

If you want to parallelize on a machine with multiple cores, you can just use (GNU) xargs, e.g.:

echo seq_[0-9][0-9].gz | xargs -n 1 -P 16 ./crunching

Meaning: xargs starts up to 16 processes in parallel of ./crunching using 1 token from stdin for each process.

You can also use split in combination with xargs.

Or you can create a simple Makefile for Job execution and call make -f mymf -j $CORES (you need temporary files for this solution).

PS: The GNU parallel manual also includes some comparisons with other tools, including xargs and make, interestingly they write:

(Very early versions of GNU parallel were coincidently implemented using make -j).

share|improve this answer
    
It is important for the asker that "all the output can be piped or stored". xargs is notoriously bad at that when run in parallel because the output can mix. –  Ole Tange Jun 21 '11 at 10:40
    
@maxschlepzig: I agree with your answer 100%, I always just use xargs. Many people just don't know GNU xargs is able to do parallel! –  TechZilla Dec 29 '11 at 19:09

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.