Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I want to parallelize the for loops of the following code. How to do this?

#!/bin/bash
N=$1 
n=$2
for (( i=1; i<=$N; i++ )); do
  min=100000000000000  //set min to some garbage value
  for (( j=1; j<=$n; j++ )); do
    val=$(/path/to/a.out)
    val2=`echo $val | bc`  

      if (( $val2 < $min )); then
        min=$val2; 
      fi
  done
  arr=("${arr[@]}" "$min")
done
share|improve this question
    
possible duplicate of Poor Man's GNU Parallel implemented in ksh? –  Gilles Dec 2 '11 at 23:47
    
See also Four tasks in parallel… how do I do that? –  Gilles Oct 31 '12 at 22:11

2 Answers 2

#!/bin/bash
# set -x # debug version
N=${1:-123}
n=${2:-45}
workers=${workers:-${3:-10}}
((workers < 1)) && ((workers = 1))
((workers > 20)) && ((workers = 20))

((min=100000000000000))  #set min to some garbage value

work() {
  for i in ${*}; do
    for (( j=1; j<=${n}; j++ )); do
      val=$(/path/to/a.out)
      val2=$(echo ${val} | bc)
      (( val2 < min )) && (( min = val2 ));
    done
    echo ${min}
    # # debug version
    # echo ${i} ${j} ${min}
  done
}

# --
arr=($(
  seq ${N} | xargs -n$[N/workers + 1] | while read i; do
    work ${i} &
  done
  wait
))
echo ${arr[*]}
# --

# # debug version
# seq ${N} | xargs -t -n$[N/workers + 1] | while read i; do
#  work ${i} &
# done
# wait

Always use workers when spawning a parameterized numbers of processes and limit the maximum number of workers that can spawn.

xargs -n | while read is a simple means of iterating lists in batches.

  • seq creates a list of numbers from 1 to N.
  • xargs -n breaks that list into N/workers+1 batches.
    • e.g. N=100 workers=10 will produces 10 lines of up to 11 numbers from 1 to 100.
  • while read i reads each line of numbers.
  • work ${i} & just calls the work function with ${i} batch of numbers.

To debug i've added commented-out debug code. Just replace the echo with the debug version and the code between # -- with its debug version and you can see how it works on batches. Uncomment set -x for a more detailed debug output which you might want to redirect to a file.

Just run the debug version with different parameters to watch how it operates:

parallel.sh 223 5 1
parallel.sh 223 5 5
parallel.sh 223 5 10
parallel.sh 223 5 20

Disclaimer: This code doesn't synchronize the min value between worker processes. Obtaining the minimum value isn't a horrible exercise. This'll probably do:

parallel.sh 223 5 20 | tr ' ' '\n' | sort -n | head -1

Or just add the same to the script itself:

echo ${arr[*]} | tr ' ' '\n' | sort -n | head -1
share|improve this answer
    
Can you explain the code? It is not clear to me how you have used worker threads here. –  podu Dec 3 '11 at 13:09
    
@prasenjit Done. I hope that's helpful. –  nicerobot Dec 3 '11 at 16:42
    
Thanks for explaining in detail –  podu Dec 4 '11 at 4:35

Using GNU Parallel:

#!/bin/bash

N=$1
n=$2

arr=($(
# Generate all combinations of 1..n and 1..N
parallel -k --tag /path/to/a.out {1} {2} '|' bc :::: <(seq $N) <(seq $n) |
  perl -ane 'BEGIN{$min=1e30} $last||=$F[0]; if($F[0] != $last) {print $min,"\n";$min=1e30;$last=$F[0]} $min = $F[2]<$min ? $F[2] : $min; END {print $min,"\n"}'
))
echo ${arr[*]}

This will run /path/to/a.out on each cpu. The output will be like:

1 1 19269
1 2 6158
1 3 2794
1 4 25104
2 1 13160
2 2 32683
2 3 12535
2 4 15197
3 1 8228
3 2 7673
3 3 8428
3 4 24463

The perl script looks at the first column and finds the minimum in the 3rd column that has the same first column.

You can install GNU Parallel simply by:

wget http://git.savannah.gnu.org/cgit/parallel.git/plain/src/parallel
chmod 755 parallel

Watch the intro videos to learn more: https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

share|improve this answer
    
The link in your last paragraph links to a facebook looking page that tells the visitor that he has won an exclusive gift (advertising or scam). There are no videos. You should remove it. –  Marco Oct 31 '12 at 7:33
    
Thanks. nd.gd has apparently vanished. Url fixed. –  Ole Tange Oct 31 '12 at 15:26

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.