22

I want to use wget (from a php script) to download image files, but don't want to download files over a certain size.

Can I limit file size with wget? If not, what is a better way?

Improve this question

5 Answers 5

Reset to default
21

If you are scripting downloads, you should consider using curl instead. Wget can parse output and recursively fetch whole sites, but curl has way more options relating to the actual download of a specific file. Here is the relevant option in the man page:

--max-filesize
Specify the maximum size (in bytes) of a file to download. If the file requested is larger than this value, the transfer will not start and curl will return with exit code 63.
NOTE: The file size is not always known prior to download, and for such files this option has no effect even if the file transfer ends up being larger than this given limit.

The note about this only working for some files is worth considering. The client is dependent on the server to report how big the file is going to be before it starts downloading. Most but certainly not all servers report this.

Improve this answer
2
  • 6
    You can combine this with a system limit to forbid curl from creating a file larger than $n bytes: (ulimit -f $(($n/512)); curl --max-filesize $n …). curl will abort with an error if the file size goes over $n/512 512-byte blocks.
    – Gilles 'SO- stop being evil'
    Commented Jul 10, 2011 at 21:22
  • You can also you httrack.
    – Vi.
    Commented Dec 30, 2013 at 1:46
9

If you want to use wget, here is a way to test the size of the file without downloading:

wget --spider $URL 2>&1 | awk '/Length/ {print $2}'

where $URL is the URL of the file you want to download, of course.

So you can condition your script based on the output. such as:

{ [ $(wget --spider $URL 2>&1 | awk '/Length/ {print $2}') -lt 20971520 ] && wget $URL; } || echo file to big

for limiting the download size to 20 MB.

(the code is ugly, for informational purposes only).

Improve this answer
3

Yes, there is a reasonable way to set a max file size with wget, the question is enought reason. So in this link there is a great patch for wget, and a wget already compiled. I used it for my reasonable reason I had, and it worked so good. I'd like they add this to de wget in linux repository.

https://yurichev.com/wget.html

Good luck.

Improve this answer
1
  • The precompiled (linux x64 patched wget 1.18) requires libnettle6, which is removed from the latest Ubuntu 20.04 packages. I found that this version [packages.debian.org/stretch/libnettle6] works great.
    – Limina102
    Commented May 1, 2021 at 15:07
2

There doesn't seem to be any reasonable way to set a max file size with wget.

Improve this answer
1
  • 2
    Gilles's approach with ulimit (see bash(1), setrlimit(2)) will also work with wget(1), though it is a bit heavy-handed.
    – sarnold
    Commented Jul 11, 2011 at 0:01
1

wget has a -Q --quota option that limits the max download amount when several URLS (recursive or input list) are given.

https://www.gnu.org/software/wget/manual/html_node/Download-Options.html

https://stackoverflow.com/a/20318140/4364036

Improve this answer
1
  • From the man page: "Note that quota will never affect downloading a single file... The quota is checked only at the end of each downloaded file, so it will never result in a partially downloaded file."
    – SArcher
    Commented Mar 14, 2023 at 20:57

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .