I'd like to use the feature of GNU parallel where it can execute the command and the list it's fed in parallel and spit it out after it's all done, however, I don't want to install GNU parallel across all of our servers.
Or perhaps a parallel version of xargs?
Is there a ksh implementation of what GNU Parallel does? In this case, it doesn't have to be done in order as GNU Parallel does -- just as long as all the output can be piped or stored. I'd also like to avoid using temporary files.
If you want to parallelize on a machine with multiple cores, you can just use (GNU) xargs, e.g.:
echo seq_[0-9][0-9].gz | xargs -n 1 -P 16 ./crunching
Meaning: xargs starts up to 16 processes in parallel of ./crunching using 1 token from stdin for each process.
You can also use split in combination with xargs.
Or you can create a simple Makefile for Job execution and call make -f mymf -j $CORES (you need temporary files for this solution).
PS: The GNU parallel manual also includes some comparisons with other tools, including xargs and make, interestingly they write:
(Very early versions of GNU parallel were coincidently implemented using make -j).
xargs is notoriously bad at that when run in parallel because the output can mix.
Jun 21, 2011 at 10:40
Look at parallel --embed which embeds GNU Parallel into the same shell script that you use it from.
parallel --embed > new_script
then edit new_script.
