2

I'm trying to mirror a site but the server only responses with gzip pages so wget won't recurse. I've searched around and there are some references to a patch to add gzip support to wget however they seem to be out of date. Is there anyway to do this? If not I was considering reverse proxying it through nginx.

  • 2
    Have you already tried sending Accept-Encoding: identity in the HTTP request header? – scai Dec 13 '12 at 8:11
  • Yes to no effect – jpiasetz Dec 13 '12 at 8:56
  • Do you have to use wget? This may be easier to do with Perl/Python/Ruby, all of which have easy-to-use http retrieval and html parsing libraries. – Gilles Dec 14 '12 at 0:07
2

You have 4 ways:

  1. wget one page, gunzip it and process it again from the html... iterate until finished:

    wget -m http://example.org/page.html
    find . -name \*gz -exec gzip -d {} \;
    find . -name \*html -exec wget -M -F {} \;</code></pre>
    

    This will be slow, but should work.

  2. Install Privoxy and configure it to uncompress the requested pages:

    +prevent-compression

    Prevent the website from compressing the data. Some websites do that, which is a problem for Privoxy when built without zlib support, since +filter and +gif-deanimate will not work on compressed data. Will slow down connections to those websites, though.

  3. Privoxy or another proxy might also be able to get the compressed pages and deliver the uncompressed copy to the client; Google for it.

  4. My wget wont send the "Accept-Encoding: gzip" header that requests gzip content... Check why yours does it. Maybe you have a proxy that is adding it? You can also use Privoxy to remove that header.

  • The wget example is likely going to be a bit more involved than that. Gzipped content doesn't automatically get a gz name. Then the subsequent find/wget might need to deal with any text/html content downloaded rather than *.html. Maybe via --adjust-extension but then recursion will be slow and you'd need to convert links which you can't because the download is gzipped... and so on. A proxy that removes encodings will be easier. Fiddler will do it. privoxy should too if it's doing any filtering but +prevent-compression just modifies the header – Matt Dec 14 '12 at 7:12
1

A belated response, but for purpose of web reference:

wget -O - $URL | zcat | wget -B $URL -i - -F -c -T 45 -k -r --page-requisites

"Known good" for documentation caching. The Timeout/Retry options (such as with -c and -T) may be useful with regards to intermittent "Link drops" in wireless networks. All options optional, of course.

  • explainshell.com/… – Tom Sep 23 '17 at 11:16
  • this could be improved by using zcat -f so that it will handle non-compressed urls too. – Tom Sep 23 '17 at 11:21
  • This looks promising, but I haven't managed to get it working - it doesn't seem to save the original (base) page (since that is in stdout only?) and it's fetching links from <a href=""> tags, not just images & scripts. any ideas? – Tom Sep 23 '17 at 11:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.