curl lacks a pause between requests, wget lacks dynamic file output names, is there a ready alternative for sequential range of file downloads?

Question

Both curl and wget offer the ability to download a sequential range of files ([1-100] in curl, {1..100} in wget) but each of them have a shortcoming: curl offers no easy way to pause between each download in the sequence. Some servers cut off downloads after several rapid downloads, and, in any case, it is polite and proper to pause between downloads anyways to be a good scraper citizen. If one wanted to, say, pause 5 seconds between each request, my understanding is there is no way to do this without additional scripting that essentially defeats the point of having the built-in support for a sequential range by making individual requests.

A solution to this is to use wget which has the handy --wait=5 flag to achieve the above desired result. Unfortunately, wget has other problems. It seems to struggle with special characters in URLs, and quotes around the URL can't be used because the range {1..100} then appears to go unrecognized. This means some manual escaping of special characters is sometimes needed. This is manageable, but annoying.

However, more importantly, wget has no support for naming the output dynamically (the -O flag is of no help here). Though curl offers the convenient -o "#1.jpg" there appears to be no way to achieve the same dynamic result in wget without, again, bypassing built-in sequential range support and making a scripted collection of single requests, or else having to rename or otherwise edit the file names after download.

This strikes me as a fairly common task: downloading a sequential range of source files, politely pausing between each request, and renaming the output dynamically. Am I missing some alternative to curl and wget that overcomes the two problems above: 1) pause between each request 2) output file names dynamically.

Answer 1

It seems to struggle with special characters in URLs, and quotes around the URL can't be used because the range {1..100} then appears to go unrecognized.

This is because this range syntax is not actually a feature of wget, it's a feature of your shell (e.g. bash), which expands the arguments before passing them to wget, compare:

$ echo abc{1..5}
abc1 abc2 abc3 abc4 abc5

or

$ ruby -e 'p ARGV' abc{1..5}
["abc1", "abc2", "abc3", "abc4", "abc5"]

If you quote the argument then the shell will not expand it:

$ echo 'abc{1..5}'
abc{1..5}

However you can quote everything except the range:

$ echo 'abc'{1..5}'def'
abc1def abc2def abc3def abc4def abc5def

However, more importantly, wget has no support for naming the output dynamically

wget has no features for dealing with ranges like this, because ranges like this are not a wget feature.

So no, it seems you can't do all of this with a single command. But you can still fit it in a oneliner:

for i in {1..100}; do curl "https://example.com/${i}.jpg" -o "output_${i}.jpg"; sleep 5; done

UNIX tools are designed to be fairly focused but easily scriptable. Some of them have grown many options to accomplish common tasks in one go, but they'll never be able to cover every use case on their own.

Answer 2

Not a direct answer to "can I pause in between requests", but more so for being a "good scraper citizen".

I've succeeded with using the --limit-rate option after I was getting "Too many requests" from the endpoint. Had to use trial and error and 50K worked for my task.

curl --limit-rate 50k "https://someURL.com/resource?p=[1-100]" -o "path\to\file_#1.txt"

https://catonmat.net/cookbooks/curl/make-curl-slow

Answer 3

With curl 7.84.0 and later, you can use the --rate argument described in this blog post:

Set the fastest allowed rate with --rate "N/U" where N is an integer number and U is a time unit. Supported units are ‘s’ (for second), ‘m’ (for minute), ‘h’ (for hour) and ‘d’ (for day, as in a 24 hour unit). U is the time unit. The default time unit, if no “/U” is provided, is number of transfers per hour.

For example, to make curl not do its requests faster than twice per minute, use --rate 2/m but if you rather want 25 per hour, then use --rate 25/h.

So to send requests no more often than once every 5 seconds (which is admittedly not the same as pausing 5 seconds between requests), you would use --rate 12/m.