I am trying to run groups of commands in parallel. There are 10 groups and each group has 3 commands. I want sequential execution within each group and parallel execution of the 10 groups. I am trying something like this, but this does not seem to work:

(command11; command12; command13 &)
(command21; command22; command23 &)
(command31; command32; command33 &)
(command41; command42; command43 &)

With GNU Parallel it would be:

parallel 'command{}1; command{}2; command{}3' ::: {1..4}

or if you prefer Command substitution:

parallel 'command{}1; command{}2; command{}3' ::: $(seq 4)

GNU Parallel has about 150 job control option which you might want to study but start with the video tutorial.


As you wish

Nearly hit. The correct syntax is:

(command11; command12; command13) &
(command21; command22; command23) &
(command31; command32; command33) &
(command41; command42; command43) &

As you wish, but better

Or, if you want inside a group the commands to be left out after a command failed, then the syntax is

command11 && command12 && command13 &
command21 && command22 && command23 &
command31 && command32 && command33 &
command41 && command42 && command43 &

Note, "&&" has a very different meaning as "&". "&&" means, that the command will run only after the previous is ready, and only if it was executed without a failure (i.e. its exit code is zero). "&" means that the command before it will be run parallel with the main execution line, and its failure or success doesn't matter.

Using the xargs tool

However, these solutions have the disadvantage, that you can't wait after all of these commands were executed. To do that, we have a little bit more trickery. Specially in your case, the required command to do this would be

for i in 1 2 3 4 5 6 7 8 9 10
  echo "command${i}1 && command${i}2 && command${i}3"
done | xargs -P 10 -l 1 bash -c

It uses the very useful xargs tool. Of course you can pipe anything into it, for example if you want to process thousands of things in 10 threads parallel, it will do.

It works on a way, that it calls the command bash -c for all lines of input, parallel, but so, that always at most 10 child processes will coincidentally run (this is done by the -P 10). The xargs command end only if all of the bash processes were executed.

Parallel tool (see other answers)

The GNU has also written a tool named parallel as well. As far I know, its syntax is a little bit more clear as of the xargss, although it has lesser features and it is not so common. Likely other answers will explain it.

NodeJs "parallel" module

If you are working in the nodejs framework, also there is a very commonly used tool (npm install parallel), which is used mainly for parallelizing automatized build tasks, but it can be used also in shellscripts easily.

It is not a very good idea in common environments, because npm packages don't integrate very well with the Unix packet-handling environment. Although its syntax are far more easy, its features are far behind all of the other solutions.

In the case of your specific problem, I would likely choose the second solution in a simple case, and the xargs-based solution in a more complex one.

  • 1
    Checking the parallel, I think it nearly so many as xargs, although also its syntax is nearly so complex, if you use for complex tasks. For easy tasks, it is much simpler. – peterh Aug 17 '17 at 17:38
  • GNU Parallel is built to be switch compatible with xargs. – Ole Tange Aug 18 '17 at 6:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.