How to disable gzip compression with wget?

Question

There are other questions asking how to enable gzip compression with wget, and lots of web pages out there telling how to do this, but I need the opposite. I'm trying to locally mirror a site, and I'm just getting the home page as a gzipped file, which in turn breaks the recursion, so I can't get the whole site.

I can gunzip that file, but that still doesn't give me a recursive download of the whole cotton-pickin' site.

How do I turn off or prevent gzipping?

EDIT: The exact command I issued is

wget --random-wait -r -p -e robots=off -U mozilla http://www.example.com --reject png,jpg,jpeg,gif  --progress=dot --wait=7

Answer 1

D'oh! I figured it out. I had put

header = Accept-Encoding: gzip,deflate

in my ~/.wgetrc some time ago, assuming that it would only affect the way data was passed across the network, never thinking that wget would be unable to read the gzipped data.

In retrospect it makes sense: this is just a header that wget allows you to use (since it allows you to use any header that a browser might pass, or any you want to make up for that matter) rather than a switch built into wget, so why would anyone expect wget to automatically handle the gzipping? It would certainly be nice if it did, but there's no reason to assume that it would.

Answer 2

If specifying --header='Accept-Encoding: gzip,deflate' won't work, maybe this one would:

wget -O- http://example.com/file.gz | gunzip -c

Or by using curl and its libz library which suppose to do automatic decompression of compressed files over HTTP (curl -V | grep libz). It has also --compress parameter.