I have a program ./pgm taking some arguments (say -a file1 -b val), which requires 2 seconds to execute. I would like to use all the processors on my machine to run this program on all the input files in parallel (about 1000). What I do now, is put all the commands

./pgm -a file1 -b 12 > out1.txt &
./pgm -a file2 -b 14 > out2.txt &
./pgm -a file3 -b 16 > out3.txt &
./pgm -a file4 -b 18 > out4.txt &

in a file, and execute this file. I thought this would use all the available processors, but the number of parallel execution is very limited.

How can I achieve this? Note that parallel command is not an option.

  • What machine is this? Is this a normal PC or some kind of server with multiple nodes?
    – terdon
    Commented Feb 18, 2014 at 15:01
  • Maybe pgm is i/o bound then extra CPU doesn't help. vmstat 1 1 in a second terminal whilst other is running for a few secconds would give you a rough idea.
    – X Tian
    Commented Feb 18, 2014 at 15:03
  • 1
    Which parallel are you referring to GNU parallel or the one from moreutils? Or both?
    – Anthon
    Commented Feb 18, 2014 at 15:57
  • Can you based on oletange.blogspot.dk/2013/04/why-not-install-gnu-parallel.html elaborate why GNU Parallel is not an option?
    – Ole Tange
    Commented Feb 18, 2014 at 19:48
  • Thanks for the answer. @OleTange: I tried but the installation failed, and I am not root.
    – wwjoze
    Commented Feb 19, 2014 at 0:10

2 Answers 2


With GNU xargs:

seq 1000 | xargs -P4 -n1  sh -c 'exec ./pgm -a "file$1" -b 12 > "out.$1"' sh &

Would run up to 4 ./pgms in parallel.

Otherwise, with pdksh/mksh/oksh:

trap : CHLD
for f in file*; do
  jobs=$(jobs | wc -l)
  if (($jobs < 4)); then
    ./pgm "$f" > out.$((++n)) &
trap - CHLD

the details of signal handling vary from one shell to the next. That trick works in pdksh and its derivatives but not in any other shell I tried. You need a shell where one can trap SIGCHLD (excludes bash), where the SIGCHLD handler is executed straight away (not blocked during a wait) (excludes ash, yash), where the SIGCHLD handling interrupts the wait (excludes ksh93 and zsh).

In shells other than bash, you could also look at approaches where jobs are started in the SIGCHLD handler.

  • Thanks for the command line using xargs. But it seems my process uses to much IO to be fully parallelized.
    – wwjoze
    Commented Feb 19, 2014 at 0:22

I was having a similar kind of issue. Since you had specified parallel is not an option, you might need to look into swift as discussed in the answer of this question.

  • Whilst this may theoretically answer the question, it would be preferable to include the essential parts of the answer here, and provide the link for reference.
    – slm
    Commented Feb 18, 2014 at 22:10
  • Sure. I will keep this in mind before posting any future answers. Thanks for pointing it out.
    – Ramesh
    Commented Feb 18, 2014 at 22:19

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .