Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I want to use wget (from a php script) to download image files, but don't want to download files over a certain size.

Can I limit file size with wget? If not, what is a better way?

share|improve this question

3 Answers 3

up vote 11 down vote accepted

If you are scripting downloads, you should consider using curl instead. Wget can parse output and recursively fetch whole sites, but curl has way more options relating to the actual download of a specific file. Here is the relevant option in the man page:

--max-filesize
Specify the maximum size (in bytes) of a file to download. If the file requested is larger than this value, the transfer will not start and curl will return with exit code 63.
NOTE: The file size is not always known prior to download, and for such files this option has no effect even if the file transfer ends up being larger than this given limit.

The note about this only working for some files is worth considering. The client is dependent on the server to report how big the file is going to be before it starts downloading. Most but certainly not all servers report this.

share|improve this answer
6  
You can combine this with a system limit to forbid curl from creating a file larger than $n bytes: (ulimit -f $(($n/512)); curl --max-filesize $n …). curl will abort with an error if the file size goes over $n/512 512-byte blocks. –  Gilles Jul 10 '11 at 21:22
    
You can also you httrack. –  Vi. Dec 30 '13 at 1:46

If you want to use wget, here is a way to test the size of the file without downloading:

wget --spider $URL 2>&1 | awk '/Length/ {print $2}'

where $URL is the URL of the file you want to download, of course.

So you can condition your script based on the output. such as:

{ [ $(wget --spider $URL 2>&1 | awk '/Length/ {print $2}') -lt 20971520 ] && wget $URL; } || echo file to big

for limiting the download size to 20 MB.

(the code is ugly, for informational purposes only).

share|improve this answer

There doesn't seem to be any reasonable way to set a max file size with wget.

share|improve this answer
2  
Gilles's approach with ulimit (see bash(1), setrlimit(2)) will also work with wget(1), though it is a bit heavy-handed. –  sarnold Jul 11 '11 at 0:01

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.