2

I have a program ./pgm taking some arguments (say -a file1 -b val), which requires 2 seconds to execute. I would like to use all the processors on my machine to run this program on all the input files in parallel (about 1000). What I do now, is put all the commands

./pgm -a file1 -b 12 > out1.txt &
./pgm -a file2 -b 14 > out2.txt &
./pgm -a file3 -b 16 > out3.txt &
./pgm -a file4 -b 18 > out4.txt &
...

in a file, and execute this file. I thought this would use all the available processors, but the number of parallel execution is very limited.

How can I achieve this? Note that parallel command is not an option.

  • What machine is this? Is this a normal PC or some kind of server with multiple nodes? – terdon Feb 18 '14 at 15:01
  • Maybe pgm is i/o bound then extra CPU doesn't help. vmstat 1 1 in a second terminal whilst other is running for a few secconds would give you a rough idea. – X Tian Feb 18 '14 at 15:03
  • 1
    Which parallel are you referring to GNU parallel or the one from moreutils? Or both? – Anthon Feb 18 '14 at 15:57
  • Can you based on oletange.blogspot.dk/2013/04/why-not-install-gnu-parallel.html elaborate why GNU Parallel is not an option? – Ole Tange Feb 18 '14 at 19:48
  • Thanks for the answer. @OleTange: I tried but the installation failed, and I am not root. – wwjoze Feb 19 '14 at 0:10
5

With GNU xargs:

seq 1000 | xargs -P4 -n1  sh -c 'exec ./pgm -a "file$1" -b 12 > "out.$1"' sh &

Would run up to 4 ./pgms in parallel.

Otherwise, with pdksh/mksh/oksh:

trap : CHLD
n=0
for f in file*; do
  jobs=$(jobs | wc -l)
  if (($jobs < 4)); then
    ./pgm "$f" > out.$((++n)) &
  else
    wait
  fi
done
trap - CHLD
wait

the details of signal handling vary from one shell to the next. That trick works in pdksh and its derivatives but not in any other shell I tried. You need a shell where one can trap SIGCHLD (excludes bash), where the SIGCHLD handler is executed straight away (not blocked during a wait) (excludes ash, yash), where the SIGCHLD handling interrupts the wait (excludes ksh93 and zsh).

In shells other than bash, you could also look at approaches where jobs are started in the SIGCHLD handler.

  • Thanks for the command line using xargs. But it seems my process uses to much IO to be fully parallelized. – wwjoze Feb 19 '14 at 0:22
1

I was having a similar kind of issue. Since you had specified parallel is not an option, you might need to look into swift as discussed in the answer of this question.

  • Whilst this may theoretically answer the question, it would be preferable to include the essential parts of the answer here, and provide the link for reference. – slm Feb 18 '14 at 22:10
  • Sure. I will keep this in mind before posting any future answers. Thanks for pointing it out. – Ramesh Feb 18 '14 at 22:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.