3

I am trying to run subscripts from a main script, but I want to make sure than no more than n subscripts run at the same time.

The following simplified example illustrates.

Each subscript creates a dummy file in RAM (/dev/shm/) with a name made of a unique timestamp, and deletes it once done.

The main script counts the number of dummy files in /dev/shm/ originating from the subscripts, and doesn't (shouldn't) launch a new subscript if 2 or more of them are already running.

However, the main script seems to ignore the while condition, and launches all 5 subscripts at once.

What's wrong with my code?

mainscript.txt

#!/bin/bash
for counter in $(seq 1 5)
do
        while [ $(ls -1 /dev/shm/|grep "script044"|wc -l) -ge 2 ]
        do
                sleep 0
        done

        xterm -e "bash script044.txt" &
done

exit

script044.txt (subscript)

#!/bin/bash

tempfilename="script044_"$(date +%Y%m%d%H%M%S)_${RANDOM}
echo > /dev/shm/${tempfilename}

for counter in $(seq 1 $(shuf -i 10-45 -n 1))
do
        sleep 1
        printf "${counter}\r"
done

rm /dev/shm/${tempfilename}

exit
3

(Convention - .txt are just plain text files. .sh files are shell script files.).

Your mainscript.txt script has a race condition. Specifically the while loop starts its next iteration before the script044.txt script is able to create the temporary file. In fact the whole loop is iterated through before any of these files get created.

A more robust way to deal with this sort of thing is to forget the temporary files and use the shell builtin wait instead:

#!/bin/bash

pid_count=0
for counter in $(seq 1 5)
do
    xterm -e "bash script044.txt" &
    if (( ++pid_count > 2 )); then
        wait -n
        ((pid_count--))
    fi
done

This increments a counter every time a subprocess is started. If the counter is greater than 3 then we wait for the next subprocess to finish. When wait returns, we then decrement the counter and go around again to start the next xterm.

You can remove all the tempfilename-related lines from the script044.txt - they are no longer needed.


As @chepner points out, the required -n option is only available in bash 4.3 or later.

|improve this answer|||||
  • 3
    Nice, but you should use $! instead of $$? – yaegashi Jul 31 '15 at 16:43
  • 1
    Note that wait -n was only introduced in bash 4.3. – chepner Jul 31 '15 at 17:14
  • @yaegashi Yes, of course! Surprisingly it seemed to work with $$ too, though that doesn't totally make sense – Digital Trauma Jul 31 '15 at 17:31
  • 1
    @yaegashi Ah - now I see it - wait -n just waits for the next sub-process to end - we don't need the pid list at all! – Digital Trauma Jul 31 '15 at 17:43
  • 1
    @mikeserv Thats the point of the -n option. Without it we wait for all background child processes to finish. With it we just wait for the next one to finish. – Digital Trauma Jul 31 '15 at 19:21
2

While it won't help if you are aiming for a shell-only solution, GNU parallel provides a sem command that can help with this exact situation.

The following (untested since I don't have your script) should run your job 5 times, but only 2 at a time, waiting to exit at the end:

LIMIT=2
for i in {1..5}; do
    sem -j $LIMIT 'term -e "bash script044.txt"'
done
sem --wait
|improve this answer|||||
1

It looks like you want IPC. Rather than looping over sleep and doing a test each time you do, you can just wait for the child process to tell when its done. This is what pipes are for.

You can have the child processes report to the parent. Open a pipe and share it with them. When they're through, they just need to let the parent know.

sub()(  trap "echo >&9" 0
        sleep 5
)
eval    "exec 9<>"<(echo);i=0
until   [ "$((i+=1))" -gt 5 ]
do      sub & read na <&9
        date +%S:%t"$i"
done

I open it using process substitution. If you can't do this in your shell, you can use instead:

mkfifo pipe; exec 9<>pipe; rm pipe; echo >&9

Now the initial echo puts one line in the pipe - in both cases. That puts you one wait process ahead from the start - which means you'll run two concurrent processes all along. This script uses date to report on the seconds between each sub() call. Here's the output:

34: 1
39: 2
39: 3
44: 4
44: 5

There. Now as you see, every 5 seconds a child dies and when it does it echo's a line to the pipe that read is currently blocking on. As soon as read find's a newline in input, it can quit what it's doing and start over.

You just need to put a trap in your script_044 which instructs the child processes to call out to father when they're through.

|improve this answer|||||
1

A generic way to accomplish this, is to use xargs

printf "%s\n" {1..5} | xargs -P2 -n1 -i xterm -e '/bin/echo Job {}; bash'

printf is only there to fill xargs with values. xargs default replace-str is {} if the -i flag is used. Please read the manpage of xargs to get more detail informations. This should work with the GNU tools and bash as shell.

xargs flags: 
-P     Run up to max-procs processes at a time.[…]
-n     Use  at most max-args arguments per command line.[…]
-i/-I  Replace occurrences of replace-str in the initial-arguments with names read from standard input.[…]
|improve this answer|||||
  • generic regarding the use not the used tool(s). – zstegi Aug 1 '15 at 0:56
0

Insert some delay just after launching a new instance of xterm...

        xterm -e "bash script044.txt" &
        sleep 0.1
|improve this answer|||||
0

Using GNU Parallel it looks like this:

seq 1 5 | parallel -j2 -N0 'xterm -e "bash script044.txt"'

GNU Parallel is a general parallelizer and makes is easy to run jobs in parallel on the same machine or on multiple machines you have ssh access to. It can often replace a for loop.

If you have 32 different jobs you want to run on 4 CPUs, a straight forward way to parallelize is to run 8 jobs on each CPU:

Simple scheduling

GNU Parallel instead spawns a new process when one finishes - keeping the CPUs active and thus saving time:

GNU Parallel scheduling

Installation

If GNU Parallel is not packaged for your distribution, you can do a personal installation, which does not require root access. It can be done in 10 seconds by doing this:

(wget -O - pi.dk/3 || curl pi.dk/3/ || fetch -o - http://pi.dk/3) | bash

For other installation options see http://git.savannah.gnu.org/cgit/parallel.git/tree/README

Learn more

See more examples: http://www.gnu.org/software/parallel/man.html

Watch the intro videos: https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

Walk through the tutorial: http://www.gnu.org/software/parallel/parallel_tutorial.html

Sign up for the email list to get support: https://lists.gnu.org/mailman/listinfo/parallel

|improve this answer|||||
  • Note: Ole Tange (the author of this post) is the author of GNU parallel. – Steven D Aug 1 '15 at 10:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.