How to optimize an ordered procedure in a bash loop?

Question

I want to download a multitude of files while also downloading them in order and assigning a custom name to them.

The code I have is this:

iterator=1;
while read p; do
    curl "$p" -o $((iterator++)).jpg;
done < ../outProfileImages;

This seems to be very slow compared to wget's "input file" mode, but I was not able to figure out a way to have a numbered custom name while also keeping the correct order of downloaded files using wget.

The reason I want custom names, with just the number of the order of the file, is mostly because the files have too large names when downloaded, but it would also be convenient to know the order just by looking at the filenames.

Answer 1

With GNU xargs, you could do:

awk '{printf "%04d.jpg\n%s\n", NR, $0}' < ../outProfileImages |
  xargs -d '\n' -rn2 -P10 curl -o

To run up to 10 of those curls in parallel.

Answer 2

Solution by @thanasisp improves performance considerably:

i=0
while read p; do
    ((i++))
    curl -s "$p" -o "${i}.jpg" &
done < ../outProfileImages;

It is important to increase i outside the command terminated with &, because this command will be executed asynchronously in a subshell. So the next iteration wouldn't use the updated value of i if this increment was happening in a subshell, in the background.

Answer 3

With GNU Parallel it looks like this:

parallel -j100 curl {} -o {#}.jpg < ../outProfileImages

This will download 100 images in parallel.