Here's roughly how I solved the same problem in an environment where parallel
wasn't available. This relies on bash features, so you need #!/bin/bash
or explicitly executing the script through bash.
MAX_CONCURRENT=50
n=0
some_command_that_outputs_urls \
| while read url
do
{
do_something_with $url
} &
PIDS="$PIDS $!"
((++n))
if test "$n" -ge "$MAX_CONCURRENT"
then
n=0
wait $PIDS
PIDS=""
fi
done
test -n "$PIDS" && wait $PIDS
You can adjust $MAX_CONCURRENT
to specify the desired (approximate) maximum number of threads. And of course, you'll replace some_command_that_outputs_urls
and do_something_with $url
with whatever happens to be appropriate in your scenario. For example, you might replace the line some_command_that_outputs_urls \
with
for i in {0800..9999}; do
for j in {001..032}; do
printf 'http://example.com/%s-%s.jpg\n' $i $j
done
done \
# ...| while read url ...
and do_something_with $url
with simply
wget $url
giving you the final result
MAX_CONCURRENT=50
n=0
for i in {0800..9999}; do
for j in {001..032}; do
printf 'http://example.com/%s-%s.jpg\n' $i $j
done
done \
| while read url
do
{
wget $url
} &
PIDS="$PIDS $!"
((++n))
if test "$n" -ge "$MAX_CONCURRENT"
then
n=0
wait $PIDS
PIDS=""
fi
done
test -n "$PIDS" && wait $PIDS
The way this works is to have a command that generates the list of (in this case) URLs on its standard output, and read that one line at a time into the while
loop (watch out for newlines!). It will spawn up to $MAX_CONCURRENT
simultaneous processes, using $n
to keep track of how many have been spawned and $PIDS
to record their process IDs. Once $MAX_CONCURRENT
processes have been spawned (note that what's really being spawned is a compound statement, so you can have multiple commands and even blocks within it), it will wait
on the spawned PIDs (this returns immediately if none of the specified PIDs are still executing) and reset its internal state, then proceed with another run.
There are several ways that this script could be improved, including better handling of reused PIDs, but it does the job I wanted it to do in the environment where it needed to run, so it's good enough for me. In my actual version, there's also a timeout in place, and it is re-executed regularly via cron, so the risk of runaway execution times is much reduced compared to this simpler version.