3

Let us consider n statements s1, s2, s3,..., sn. Each statement has infinite steps of execution and I want to run all in parallel.

my shell script looks like

s1 
s2 
s3
.
.
.
sn

How to change my code so that the interleaving happen?

4

2 Answers 2

5

It sounds like asking each statement to run in the background could achieve what you want:

s1 &
s2 &
...
sn &
wait

(The wait command makes the script run in the foreground until all background tasks have completed.)

If you need more control, look into using GNU parallel.

1
  • I also think this is the good method for launching tasks & then you can use 6 user reserved signals if you need to trap returns or user actions to manage tasks after they are launched.
    – francois P
    Jan 17, 2018 at 17:30
4

GNU Parallel is made for that:

cat file_with_commands | parallel

It defaults to one job per CPU core. If you want more than you have cores:

parallel -j200 < file_with_commands

If your commands are very similar:

parallel s{} ::: {1..100}

will run s1, s2 ... s100 in parallel.

GNU Parallel is a general parallelizer and makes is easy to run jobs in parallel on the same machine or on multiple machines you have ssh access to.

If you have 32 different jobs you want to run on 4 CPUs, a straight forward way to parallelize is to run 8 jobs on each CPU:

Simple scheduling

GNU Parallel instead spawns a new process when one finishes - keeping the CPUs active and thus saving time:

GNU Parallel scheduling

Installation

For security reasons you should install GNU Parallel with your package manager, but if GNU Parallel is not packaged for your distribution, you can do a personal installation, which does not require root access. It can be done in 10 seconds by doing this:

(wget -O - pi.dk/3 || curl pi.dk/3/ || fetch -o - http://pi.dk/3) | bash

For other installation options see http://git.savannah.gnu.org/cgit/parallel.git/tree/README

Learn more

See more examples: http://www.gnu.org/software/parallel/man.html

Watch the intro videos: https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

Walk through the tutorial: http://www.gnu.org/software/parallel/parallel_tutorial.html

Sign up for the email list to get support: https://lists.gnu.org/mailman/listinfo/parallel

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.