Download several files with wget in parallel

Question

I have the following bash script:

for i in {0800..9999}; do
    for j in {001..032}; do
        wget http://example.com/"$i-$j".jpg
    done
done

All photos are exist and in fact each iteration does not depend from another. How to parallelize it with possibility of control the number of threads?

Answer 1

Confiq's answer is a good one for small i and j. However, given the size of i and j in your question, you will likely want to limit the overall number of processes spawned. You can do this with the parallel command or some versions of xargs. For example, using an xargs that supports the -P flag you could parallelize your inner loop as follows:

for i in {0800..9999}; do
  echo {001..032} | xargs -n 1 -P 8 -I{} wget http://example.com/"$i-{}".jpg
done

GNU parallel has a large number of features for when you need more sophisticated behavior and makes it easy to parallelize over both parameters:

parallel -a <(seq 0800 9999) -a <(seq 001 032) -P 8 wget http://example.com/{1}-{2}.jpg

Answer 2

Here's a very simplistic approach. Limits threads to 10 in this example.

for i in {0800..9999}; do
    for j in {001..032}; do
        wget http://example.com/"$i-$j".jpg &
        while test $(jobs -p|wc -w) -ge 10; do sleep 0.1 ; done
    done
done

Answer 3

for i in {1..3}; do
    for j in {10..20}; do
        (wget http://example.com/"$i-$j".jpg &)
    done
done

I even tested it...

Answer 4

Here's roughly how I solved the same problem in an environment where parallel wasn't available. This relies on bash features, so you need #!/bin/bash or explicitly executing the script through bash.

MAX_CONCURRENT=50
n=0
some_command_that_outputs_urls \
| while read url
do
    {
        do_something_with $url
    } &
    PIDS="$PIDS $!"

    ((++n))
    if test "$n" -ge "$MAX_CONCURRENT"
    then
        n=0
        wait $PIDS
        PIDS=""
    fi
done
test -n "$PIDS" && wait $PIDS

You can adjust $MAX_CONCURRENT to specify the desired (approximate) maximum number of threads. And of course, you'll replace some_command_that_outputs_urls and do_something_with $url with whatever happens to be appropriate in your scenario. For example, you might replace the line some_command_that_outputs_urls \ with

for i in {0800..9999}; do
    for j in {001..032}; do
        printf 'http://example.com/%s-%s.jpg\n' $i $j
    done
done \
# ...| while read url ...

and do_something_with $url with simply

wget $url

giving you the final result

MAX_CONCURRENT=50
n=0
for i in {0800..9999}; do
    for j in {001..032}; do
        printf 'http://example.com/%s-%s.jpg\n' $i $j
    done
done \
| while read url
do
    {
        wget $url
    } &
    PIDS="$PIDS $!"

    ((++n))
    if test "$n" -ge "$MAX_CONCURRENT"
    then
        n=0
        wait $PIDS
        PIDS=""
    fi
done
test -n "$PIDS" && wait $PIDS

The way this works is to have a command that generates the list of (in this case) URLs on its standard output, and read that one line at a time into the while loop (watch out for newlines!). It will spawn up to $MAX_CONCURRENT simultaneous processes, using $n to keep track of how many have been spawned and $PIDS to record their process IDs. Once $MAX_CONCURRENT processes have been spawned (note that what's really being spawned is a compound statement, so you can have multiple commands and even blocks within it), it will wait on the spawned PIDs (this returns immediately if none of the specified PIDs are still executing) and reset its internal state, then proceed with another run.

There are several ways that this script could be improved, including better handling of reused PIDs, but it does the job I wanted it to do in the environment where it needed to run, so it's good enough for me. In my actual version, there's also a timeout in place, and it is re-executed regularly via cron, so the risk of runaway execution times is much reduced compared to this simpler version.