I want to parallelize a for loop where the number of iterations in the loop can be very large such as 10^6. So ,it will be better if I can create threads rather than process. How to do it? The code is as follows


for (( i=0; i < $N; i++ )); do

    ./random >> output /* in each iteration one random number is appended to
                          a file "output" */                                

  • Ever considered programming with parallel processing APIs? You might be interested in OpenMP, MPICH, or generally MPI – martin Dec 4 '11 at 18:57
  • Can you tell us a little more about the "why" for the random numbers? What will the data be used for, and where is this random data coming from? If you truly just need random data in a file, there may be better options to accomplish that. – Jodie C Mar 12 '12 at 1:34
  1. You can't spawn threads from a shell.
  2. You don't want to write to the same file from multiple processes.
  3. If all your random program does is generate a single number,
    • it should be fast enough that your loop will be io bound.
    • if you can, you should edit it to take an argument and print that many numbers.
    • if the actual execution is the bottleneck, you should rethink how to generate the numbers. Perhaps post the code to Code Review.

If you really really still want to do it this way, do it in chunks:

for i in {0..9}; do
    for ((j = 1; j < $N/10; j++)); do
    done > tmp$i &
for i in {0..9}; do
    wait ${pid[$i]}
cat tmp{0..9} >> outfile

I am not aware of any shell that allows you to manually create new threads, usually you can only utilise the existing threads in the current shell (or create subshells, which are indeed new processes). Use python or another language instead.

Even if you could, I really would not recommend using a shell script for something on this scale. The performance loss compared to a compiled language when running 10^6 iterations could be substantial.


The problem, as always, is resource contention. You need to limit the number of processes/threads that are running at the same time.

Also, having concurrent threads depends a LOT on what type of processing will be performed. It is much better if there is a good mix of programming: memory, cpu, i/o, etc. If all the sub-processes will be doing all cpu, then having 10-20 running at once will not speed up anything. You could try to farm the processing out to other machines; e.g. using ssh to start calls on other machines and get the results back.

A quick and dirty first pass may be something like:

N=$1 # max number to iterate on
shift # rest of command line is the command to run: e.g. "./random"
maxcount=10 # maximum in the pool
curcount=0  # how many currently in the pool
reaper () {
    curcount=`expr $curcount - 1`
spawnnext () {
    while [ $curcount -ge $maxcount ]; do
        sleep 1 # wait for a free slot in the pool
    echo $n
    "$@" &
    curcount=`expr $curcount + 1`

trap 'reaper' CHLD

for (( i=0; i < $N; i++)); do
    spawnnext $i "$@"

Note; I haven't tested this; just wrote it up on the fly.

But I would do this in a higher-level, better performing language such as Python: https://stackoverflow.com/questions/3033952/python-thread-pool-similar-to-the-multiprocessing-pool


If your task is simply to generate random numbers:

perl -e 'for($t=0;$t<1000000;$t++) { print int(rand()*1000),"\n" }'

If your task is really something else you can use GNU Parallel:

parallel ./random :::: <(seq 1000000) > output

You can install GNU Parallel simply by:

wget http://git.savannah.gnu.org/cgit/parallel.git/plain/src/parallel
chmod 755 parallel

Watch the intro videos to learn more: http://pi.dk/1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.