Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I want to parallelize a for loop where the number of iterations in the loop can be very large such as 10^6. So ,it will be better if I can create threads rather than process. How to do it? The code is as follows

N=$1

for (( i=0; i < $N; i++ )); do

    ./random >> output /* in each iteration one random number is appended to
                          a file "output" */                                

done
share|improve this question
    
Ever considered programming with parallel processing APIs? You might be interested in OpenMP, MPICH, or generally MPI –  martin Dec 4 '11 at 18:57
    
Can you tell us a little more about the "why" for the random numbers? What will the data be used for, and where is this random data coming from? If you truly just need random data in a file, there may be better options to accomplish that. –  JodieC Mar 12 '12 at 1:34

4 Answers 4

  1. You can't spawn threads from a shell.
  2. You don't want to write to the same file from multiple processes.
  3. If all your random program does is generate a single number,
    • it should be fast enough that your loop will be io bound.
    • if you can, you should edit it to take an argument and print that many numbers.
    • if the actual execution is the bottleneck, you should rethink how to generate the numbers. Perhaps post the code to Code Review.

If you really really still want to do it this way, do it in chunks:

for i in {0..9}; do
    for ((j = 1; j < $N/10; j++)); do
        ./random
    done > tmp$i &
    pid[$i]=$?
done
for i in {0..9}; do
    wait ${pid[$i]}
done
cat tmp{0..9} >> outfile
share|improve this answer
    
+1 for point #2 –  hhaamu Dec 3 '11 at 19:15

I am not aware of any shell that allows you to manually create new threads, usually you can only utilise the existing threads in the current shell (or create subshells, which are indeed new processes). Use python or another language instead.

Even if you could, I really would not recommend using a shell script for something on this scale. The performance loss compared to a compiled language when running 10^6 iterations could be substantial.

share|improve this answer

The problem, as always, is resource contention. You need to limit the number of processes/threads that are running at the same time.

Also, having concurrent threads depends a LOT on what type of processing will be performed. It is much better if there is a good mix of programming: memory, cpu, i/o, etc. If all the sub-processes will be doing all cpu, then having 10-20 running at once will not speed up anything. You could try to farm the processing out to other machines; e.g. using ssh to start calls on other machines and get the results back.

A quick and dirty first pass may be something like:

N=$1 # max number to iterate on
shift # rest of command line is the command to run: e.g. "./random"
maxcount=10 # maximum in the pool
curcount=0  # how many currently in the pool
reaper () {
    wait
    curcount=`expr $curcount - 1`
}
spawnnext () {
    n=$1
    shift
    while [ $curcount -ge $maxcount ]; do
        sleep 1 # wait for a free slot in the pool
    done
    echo $n
    "$@" &
    curcount=`expr $curcount + 1`
}

trap 'reaper' CHLD

for (( i=0; i < $N; i++)); do
    spawnnext $i "$@"
done

Note; I haven't tested this; just wrote it up on the fly.

But I would do this in a higher-level, better performing language such as Python: http://stackoverflow.com/questions/3033952/python-thread-pool-similar-to-the-multiprocessing-pool

share|improve this answer

If your task is simply to generate random numbers:

perl -e 'for($t=0;$t<1000000;$t++) { print int(rand()*1000),"\n" }'

If your task is really something else you can use GNU Parallel:

parallel ./random :::: <(seq 1000000) > output

You can install GNU Parallel simply by:

wget http://git.savannah.gnu.org/cgit/parallel.git/plain/src/parallel
chmod 755 parallel

Watch the intro videos to learn more: http://pi.dk/1

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.