Tell me more ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.
up vote 1 down vote favorite

I'm trying to mirror a site but the server only responses with gzip pages so wget won't recurse. I've searched around and there are some references to a patch to add gzip support to wget however they seem to be out of date. Is there anyway to do this? If not I was considering reverse proxying it through nginx.

share|improve this question
2  
Have you already tried sending Accept-Encoding: identity in the HTTP request header? – scai Dec 13 '12 at 8:11
Yes to no effect – jpiasetz Dec 13 '12 at 8:56
Do you have to use wget? This may be easier to do with Perl/Python/Ruby, all of which have easy-to-use http retrieval and html parsing libraries. – Gilles Dec 14 '12 at 0:07

1 Answer

up vote 1 down vote

You have 2^H4 ways:

  1. wget one page, gunzip it and process it again from the html... iterate until finish:
wget -m http://example.org/page.html
find . -name \*gz -exec gzip -d {} \;
find . -name \*html -exec wget -M -F {} \;

this will be slow, but should work

  1. install privoxy and configure it to request uncompress the pages
+prevent-compression
Prevent the website from compressing the data. Some websites do that, which is a problem for Privoxy when built without zlib support, since +filter and +gif-deanimate will not work on compressed data. Will slow down connections to those websites, though.

  1. privoxy or another proxy might also be able to get the compressed pages and deliver the uncompressed copy to the client; Google for it.

  2. my wget wont send the "Accept-Encoding: gzip" header that requests gzip content... Check why yours does it. Maybe you have a proxy that is adding it? You can also use privoxy to remove that header

share|improve this answer
The wget example is likely going to be a bit more involved than that. Gzipped content doesn't automatically get a gz name. Then the subsequent find/wget might need to deal with any text/html content downloaded rather than *.html. Maybe via --adjust-extension but then recursion will be slow and you'd need to convert links which you can't because the download is gzipped... and so on. A proxy that removes encodings will be easier. Fiddler will do it. privoxy should too if it's doing any filtering but +prevent-compression just modifies the header – mindthemonkey Dec 14 '12 at 7:12

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.