6

I have the following bash script:

for i in {0800..9999}; do
    for j in {001..032}; do
        wget http://example.com/"$i-$j".jpg
    done
done

All photos are exist and in fact each iteration does not depend from another. How to parallelize it with possibility of control the number of threads?

20

Confiq's answer is a good one for small i and j. However, given the size of i and j in your question, you will likely want to limit the overall number of processes spawned. You can do this with the parallel command or some versions of xargs. For example, using an xargs that supports the -P flag you could parallelize your inner loop as follows:

for i in {0800..9999}; do
  echo {001..032} | xargs -n 1 -P 8 -I{} wget http://example.com/"$i-{}".jpg
done

GNU parallel has a large number of features for when you need more sophisticated behavior and makes it easy to parallelize over both parameters:

parallel -a <(seq 0800 9999) -a <(seq 001 032) -P 8 wget http://example.com/{1}-{2}.jpg
3

Here's a very simplistic approach. Limits threads to 10 in this example.

for i in {0800..9999}; do
    for j in {001..032}; do
        wget http://example.com/"$i-$j".jpg &
        while test $(jobs -p|wc -w) -ge 10; do sleep 0.1 ; done
    done
done
3
for i in {1..3}; do
    for j in {10..20}; do
        (wget http://example.com/"$i-$j".jpg &)
    done
done

I even tested it...

  • 2
    It's too many threads I think. It should be possible to limit the number of threads – danielleontiev Jul 30 '17 at 21:14
  • 2
    well, that's not your original question... maybe you should update the question then? – confiq Jul 30 '17 at 21:15
  • 1
    You have modified the original question. As per the original, your solution spawns nearly 300000 processes which is of course ridiculous. – pipe Jul 31 '17 at 8:37
  • @pipe: if you would read comments, you would see that OP has edited the question – confiq Jul 31 '17 at 13:24
  • The original question has an outer loop from 800 to 9999 and an inner loop from 1 to 32. You changed that to something much smaller in order for your solution to work. – pipe Jul 31 '17 at 13:40
2

Here's roughly how I solved the same problem in an environment where parallel wasn't available. This relies on bash features, so you need #!/bin/bash or explicitly executing the script through bash.

MAX_CONCURRENT=50
n=0
some_command_that_outputs_urls \
| while read url
do
    {
        do_something_with $url
    } &
    PIDS="$PIDS $!"

    ((++n))
    if test "$n" -ge "$MAX_CONCURRENT"
    then
        n=0
        wait $PIDS
        PIDS=""
    fi
done
test -n "$PIDS" && wait $PIDS

You can adjust $MAX_CONCURRENT to specify the desired (approximate) maximum number of threads. And of course, you'll replace some_command_that_outputs_urls and do_something_with $url with whatever happens to be appropriate in your scenario. For example, you might replace the line some_command_that_outputs_urls \ with

for i in {0800..9999}; do
    for j in {001..032}; do
        printf 'http://example.com/%s-%s.jpg\n' $i $j
    done
done \
# ...| while read url ...

and do_something_with $url with simply

wget $url

giving you the final result

MAX_CONCURRENT=50
n=0
for i in {0800..9999}; do
    for j in {001..032}; do
        printf 'http://example.com/%s-%s.jpg\n' $i $j
    done
done \
| while read url
do
    {
        wget $url
    } &
    PIDS="$PIDS $!"

    ((++n))
    if test "$n" -ge "$MAX_CONCURRENT"
    then
        n=0
        wait $PIDS
        PIDS=""
    fi
done
test -n "$PIDS" && wait $PIDS

The way this works is to have a command that generates the list of (in this case) URLs on its standard output, and read that one line at a time into the while loop (watch out for newlines!). It will spawn up to $MAX_CONCURRENT simultaneous processes, using $n to keep track of how many have been spawned and $PIDS to record their process IDs. Once $MAX_CONCURRENT processes have been spawned (note that what's really being spawned is a compound statement, so you can have multiple commands and even blocks within it), it will wait on the spawned PIDs (this returns immediately if none of the specified PIDs are still executing) and reset its internal state, then proceed with another run.

There are several ways that this script could be improved, including better handling of reused PIDs, but it does the job I wanted it to do in the environment where it needed to run, so it's good enough for me. In my actual version, there's also a timeout in place, and it is re-executed regularly via cron, so the risk of runaway execution times is much reduced compared to this simpler version.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.