1

I want to download a multitude of files while also downloading them in order and assigning a custom name to them.

The code I have is this:

iterator=1;
while read p; do
    curl "$p" -o $((iterator++)).jpg;
done < ../outProfileImages;

This seems to be very slow compared to wget's "input file" mode, but I was not able to figure out a way to have a numbered custom name while also keeping the correct order of downloaded files using wget.

The reason I want custom names, with just the number of the order of the file, is mostly because the files have too large names when downloaded, but it would also be convenient to know the order just by looking at the filenames.

4
  • Have you tried to send the curl commands to the background? But do the increament before that, or else i could stay 1. i=0 and into the loop: ((i++)) and curl ... ${i}.jpg & (and be sure that you don't send too many requests...)
    – thanasisp
    Oct 30, 2020 at 20:46
  • @thanasisp Many thanks for this! It worked! Will update question with the changed source code
    – kornoborno
    Oct 30, 2020 at 21:03
  • @thanasisp You may take the credit for it then. I am just a guest here. Thanks!
    – kornoborno
    Oct 30, 2020 at 21:05
  • We are all guests, thank you too for your good will. I suggest you, whenever you can, to write it as an answer, (I will probably add a small edit for the increament). This is acceptable and could help some future reader.
    – thanasisp
    Oct 30, 2020 at 21:13

3 Answers 3

3

With GNU xargs, you could do:

awk '{printf "%04d.jpg\n%s\n", NR, $0}' < ../outProfileImages |
  xargs -d '\n' -rn2 -P10 curl -o

To run up to 10 of those curls in parallel.

2

Solution by @thanasisp improves performance considerably:

i=0
while read p; do
    ((i++))
    curl -s "$p" -o "${i}.jpg" &
done < ../outProfileImages;

It is important to increase i outside the command terminated with &, because this command will be executed asynchronously in a subshell. So the next iteration wouldn't use the updated value of i if this increment was happening in a subshell, in the background.

1

With GNU Parallel it looks like this:

parallel -j100 curl {} -o {#}.jpg < ../outProfileImages

This will download 100 images in parallel.

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.