Can one limit the max file size to download using wget?

Question

I want to use wget (executed from a php script) to download image files, but don't want to download files over a certain size.

Can I limit file size with wget? If not, what is a better way to do this?

Answer 1

If you are scripting downloads, you should consider using curl instead. Wget can parse output and recursively fetch whole sites, but curl has way more options relating to the actual download of a specific file. Here is the relevant option in the man page:

--max-filesize
Specify the maximum size (in bytes) of a file to download. If the file requested is larger than this value, the transfer will not start and curl will return with exit code 63.
NOTE: The file size is not always known prior to download, and for such files this option has no effect even if the file transfer ends up being larger than this given limit.

The note about this only working for some files is worth considering. The client is dependent on the server to report how big the file is going to be before it starts downloading. Most but certainly not all servers report this.

Answer 2

If you want to use wget, here is a way to test the size of the file without downloading:

wget --spider $URL 2>&1 | awk '/Length/ {print $2}'

where $URL is the URL of the file you want to download, of course.

So you can condition your script based on the output. such as:

{ [ $(wget --spider $URL 2>&1 | awk '/Length/ {print $2}') -lt 20971520 ] && wget $URL; } || echo file to big

for limiting the download size to 20 MB.

(the code is ugly, for informational purposes only).

Answer 3

There doesn't seem to be any reasonable way to set a max file size with wget.