There are other questions asking how to enable gzip compression with wget, and lots of web pages out there telling how to do this, but I need the opposite. I'm trying to locally mirror a site, and I'm just getting the home page as a gzipped file, which in turn breaks the recursion, so I can't get the whole site.

I can gunzip that file, but that still doesn't give me a recursive download of the whole cotton-pickin' site.

How do I turn off or prevent gzipping?

EDIT: The exact command I issued is

wget --random-wait -r -p -e robots=off -U mozilla http://www.example.com --reject png,jpg,jpeg,gif  --progress=dot --wait=7
  • 1
    Also, the manpage has no mention of gzipping at all.
    – iconoclast
    Mar 16 '13 at 1:15
  • 1
    wget does recursive downloading on its own, and I've never seen it to gzip (or otherwise pack) anything. Exactly what commands are you issuing? You might need to edit out parts of the URL for privacy, edit that into the question.
    – vonbrand
    Mar 16 '13 at 1:21

D'oh! I figured it out. I had put

header = Accept-Encoding: gzip,deflate

in my ~/.wgetrc some time ago, assuming that it would only affect the way data was passed across the network, never thinking that wget would be unable to read the gzipped data.

In retrospect it makes sense: this is just a header that wget allows you to use (since it allows you to use any header that a browser might pass, or any you want to make up for that matter) rather than a switch built into wget, so why would anyone expect wget to automatically handle the gzipping? It would certainly be nice if it did, but there's no reason to assume that it would.


If specifying --header='Accept-Encoding: gzip,deflate' won't work, maybe this one would:

wget -O- http://example.com/file.gz | gunzip -c

Or by using curl and its libz library which suppose to do automatic decompression of compressed files over HTTP (curl -V | grep libz). It has also --compress parameter.

  • 3
    This may be useful, but it isn't quite aimed at what I'm actually trying to solve with this question. I'm trying to disable gzip, because it's preventing recursion. This answer doesn't solve that.
    – iconoclast
    May 9 '15 at 1:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.