Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

There are other questions asking how to enable gzip compression with wget, and lots of web pages out there telling how to do this, but I need the opposite. I'm trying to locally mirror a site, and I'm just getting the home page as a gzipped file, which in turn breaks the recursion, so I can't get the whole site.

I can gunzip that file, but that still doesn't give me a recursive download of the whole cotton-pickin' site.

How do I turn off or prevent gzipping?

EDIT: The exact command I issued is

wget --random-wait -r -p -e robots=off -U mozilla http://www.example.com --reject png,jpg,jpeg,gif  --progress=dot --wait=7
share|improve this question
1  
Also, the manpage has no mention of gzipping at all. –  iconoclast Mar 16 '13 at 1:15
    
wget does recursive downloading on its own, and I've never seen it to gzip (or otherwise pack) anything. Exactly what commands are you issuing? You might need to edit out parts of the URL for privacy, edit that into the question. –  vonbrand Mar 16 '13 at 1:21

1 Answer 1

up vote 1 down vote accepted

D'oh! I figured it out. I had put

header = Accept-Encoding: gzip,deflate

in my ~/.wgetrc some time ago, assuming that it would only affect the way data was passed across the network, never thinking that wget would be unable to read the gzipped data.

In retrospect it makes sense: this is just a header that wget allows you to use (since it allows you to use any header that a browser might pass, or any you want to make up for that matter) rather than a switch built into wget, so why would anyone expect wget to automatically handle the gzipping? It would certainly be nice if it did, but there's no reason to assume that it would.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.