2

Both curl and wget offer the ability to download a sequential range of files ([1-100] in curl, {1..100} in wget) but each of them have a shortcoming: curl offers no easy way to pause between each download in the sequence. Some servers cut off downloads after several rapid downloads, and, in any case, it is polite and proper to pause between downloads anyways to be a good scraper citizen. If one wanted to, say, pause 5 seconds between each request, my understanding is there is no way to do this without additional scripting that essentially defeats the point of having the built-in support for a sequential range by making individual requests.

A solution to this is to use wget which has the handy --wait=5 flag to achieve the above desired result. Unfortunately, wget has other problems. It seems to struggle with special characters in URLs, and quotes around the URL can't be used because the range {1..100} then appears to go unrecognized. This means some manual escaping of special characters is sometimes needed. This is manageable, but annoying.

However, more importantly, wget has no support for naming the output dynamically (the -O flag is of no help here). Though curl offers the convenient -o "#1.jpg" there appears to be no way to achieve the same dynamic result in wget without, again, bypassing built-in sequential range support and making a scripted collection of single requests, or else having to rename or otherwise edit the file names after download.

This strikes me as a fairly common task: downloading a sequential range of source files, politely pausing between each request, and renaming the output dynamically. Am I missing some alternative to curl and wget that overcomes the two problems above: 1) pause between each request 2) output file names dynamically.

3
  • 1
    aria2 might be your friend :-)
    – pLumo
    Jul 27 '20 at 11:45
  • This looks very interesting! Does it support ranges of sequential urls (like [1-10] in a url)? I couldn't tell from the documentation. I also didn't see a flag like -w in wget to pause between requests but I saw an open feature request for it on github. Any thoughts @pLumo? Jul 27 '20 at 19:21
  • 1
    No sorry, cannot tell, otherwise I would posted it as an answer ;-) I just know that aria is a great tool that can do lots of good stuff.
    – pLumo
    Jul 27 '20 at 19:24
10

It seems to struggle with special characters in URLs, and quotes around the URL can't be used because the range {1..100} then appears to go unrecognized.

This is because this range syntax is not actually a feature of wget, it's a feature of your shell (e.g. bash), which expands the arguments before passing them to wget, compare:

$ echo abc{1..5}
abc1 abc2 abc3 abc4 abc5

or

$ ruby -e 'p ARGV' abc{1..5}
["abc1", "abc2", "abc3", "abc4", "abc5"]

If you quote the argument then the shell will not expand it:

$ echo 'abc{1..5}'
abc{1..5}

However you can quote everything except the range:

$ echo 'abc'{1..5}'def'
abc1def abc2def abc3def abc4def abc5def

However, more importantly, wget has no support for naming the output dynamically

wget has no features for dealing with ranges like this, because ranges like this are not a wget feature.

So no, it seems you can't do all of this with a single command. But you can still fit it in a oneliner:

for i in {1..100}; do curl "https://example.com/${i}.jpg" -o "output_${i}.jpg"; sleep 5; done

UNIX tools are designed to be fairly focused but easily scriptable. Some of them have grown many options to accomplish common tasks in one go, but they'll never be able to cover every use case on their own.

0
4

Not a direct answer to "can I pause in between requests", but more so for being a "good scraper citizen".

I've succeeded with using the --limit-rate option after I was getting "Too many requests" from the endpoint. Had to use trial and error and 50K worked for my task.

curl --limit-rate 50k "https://someURL.com/resource?p=[1-100]" -o "path\to\file_#1.txt"

https://catonmat.net/cookbooks/curl/make-curl-slow

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.