15

I want to use wget (from a php script) to download image files, but don't want to download files over a certain size.

Can I limit file size with wget? If not, what is a better way?

15

If you are scripting downloads, you should consider using curl instead. Wget can parse output and recursively fetch whole sites, but curl has way more options relating to the actual download of a specific file. Here is the relevant option in the man page:

--max-filesize
Specify the maximum size (in bytes) of a file to download. If the file requested is larger than this value, the transfer will not start and curl will return with exit code 63.
NOTE: The file size is not always known prior to download, and for such files this option has no effect even if the file transfer ends up being larger than this given limit.

The note about this only working for some files is worth considering. The client is dependent on the server to report how big the file is going to be before it starts downloading. Most but certainly not all servers report this.

2
  • 6
    You can combine this with a system limit to forbid curl from creating a file larger than $n bytes: (ulimit -f $(($n/512)); curl --max-filesize $n …). curl will abort with an error if the file size goes over $n/512 512-byte blocks. – Gilles 'SO- stop being evil' Jul 10 '11 at 21:22
  • You can also you httrack. – Vi. Dec 30 '13 at 1:46
6

If you want to use wget, here is a way to test the size of the file without downloading:

wget --spider $URL 2>&1 | awk '/Length/ {print $2}'

where $URL is the URL of the file you want to download, of course.

So you can condition your script based on the output. such as:

{ [ $(wget --spider $URL 2>&1 | awk '/Length/ {print $2}') -lt 20971520 ] && wget $URL; } || echo file to big

for limiting the download size to 20 MB.

(the code is ugly, for informational purposes only).

1

There doesn't seem to be any reasonable way to set a max file size with wget.

1
  • 2
    Gilles's approach with ulimit (see bash(1), setrlimit(2)) will also work with wget(1), though it is a bit heavy-handed. – sarnold Jul 11 '11 at 0:01
1

Yes, there is a reasonable way to set a max file size with wget, the question is enought reason. So in this link there is a great patch for wget, and a wget already compiled. I used it for my reasonable reason I had, and it worked so good. I'd like they add this to de wget in linux repository.

https://yurichev.com/wget.html

Good luck.

0

wget has a -Q --quota option that limits the max download amount when several URLS (recursive or input list) are given.

https://www.gnu.org/software/wget/manual/html_node/Download-Options.html

https://stackoverflow.com/a/20318140/4364036

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.