Sign up ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. It's 100% free, no registration required.

I stumbled upon this website which talk about this.

So when downloading whole website by getting the gzipped version, what is the right command?

I've tested this command, but I dont know if wget really getting the gzipped version:

wget --header="accept-encoding: gzip" -m -r -q -R gif,png,jpg,jpeg,GIF,PNG,JPG,JPEG,js,rss,xml,feed,.tar.gz,.zip,rar,.rar,.php,.txt -t 1
share|improve this question
You say you've tested that command, but @EightBitTony's answer below seems to say that what you would get out of that would be a gzip file of the first hit without any recursion through site the site for more files. Was that the result you got? – Caleb Jun 17 '11 at 14:59 is gzip-compressed, and this command recurse the entire site. I've tested this command on other website and it recurse the entire site too. Thats why I a bit confused whether it really download the gzipped version or not – jomnana Jun 17 '11 at 15:04

2 Answers 2

up vote 8 down vote accepted

If you request gzip'ed content (using the accept-encoding: gzip header, which is correct), then it's my understanding that wget can't then read the content. So you will end up with a single, gzipped file on disk, for the first page you hit, but no other content.

i.e. you can't use wget to request gzipped content and to recurse the entire site at the same time.

I think there's a patch that allows wget to support this function but it's not in the default distribution version.

If you include the -S flag you can tell if the web server is responding with the correct type of content. For example,

wget -S --header="accept-encoding: gzip"
--2011-06-17 16:06:46--
Resolving (,,
Connecting to (||:80... connected.
HTTP request sent, awaiting response...
  HTTP/1.1 200 OK
  Server: nginx
  Date: Fri, 17 Jun 2011 15:06:47 GMT
  Content-Type: text/html; charset=UTF-8
  Connection: close
  Vary: Accept-Encoding
  Last-Modified: Fri, 17 Jun 2011 15:04:57 +0000
  Cache-Control: max-age=190, must-revalidate
  Vary: Cookie
  X-hacker: If you're reading this, you should visit and apply to join the fun, mention this header.
  Link: <>; rel=shortlink
  X-nananana: Batcache
  Content-Encoding: gzip
Length: unspecified [text/html]

The content encoding clearly states gzip, however for (currently),

wget -S --header="accept-encoding: gzip"
--2011-06-17 16:12:55--
Resolving (
Connecting to (||:80... connected.
HTTP request sent, awaiting response...
  HTTP/1.1 200 OK
  Date: Fri, 17 Jun 2011 15:12:56 GMT
  Server: Apache
  Set-Cookie: TMog=B6HFCs2H20kA1I4N;; path=/; expires=Sat, 22-Sep-12 14:19:35 GMT
  Set-Cookie: Mint=B6HFCs2H20kA1I4N;; path=/
  Set-Cookie: zBT=1;; path=/
  Vary: *
  PRAGMA: no-cache
  Cache-Control: max-age=-3600
  Expires: Fri, 17 Jun 2011 14:12:56 GMT
  Connection: close
  Content-Type: text/html
Length: unspecified [text/html]

It's returning text/html.

Because some older browsers still have issues with gzip encoded content, many sites only enable it based on browser identification. They often turn it off by default and only turn it one when they know the browser can support it - and they usually don't include wget in that list. This means you may find wget never returns gzip content even if the site appears to do so for your browser.

share|improve this answer
But I got bunch of files, and not a single gzipped file... or does my wget version is different? (using Ubuntu 11.04) – jomnana Jun 17 '11 at 15:07
If you use -S you can see the headers returned from the server, and when you do that against you can clearly see it's returning html, not gzip content. wget -S --header="accept-encoding: gzip" Content-Type: text/html – EightBitTony Jun 17 '11 at 15:09
Because not all browsers support gzip encoding (IE has major issues), many websites only enable gzip encoding on a per browser basis and don't bother doing it for wget. That probably explains why doesn't gzip when asked by wget. But it doesn't fix the main issue that (AFAIK) wget can't recurse gzipped content. – EightBitTony Jun 17 '11 at 15:15
Thanks for the great explanation. – jomnana Jun 17 '11 at 15:29
Just tried this: the wget output is still Content-Type: text/html; charset=UTF-8, but there's also Content-Encoding: gzip. It wouldn't be transparent compression if using it forced the MIME type of everything to gzip... I ran strace -s 128 wget ... to actually see some of the bytes read from the socket / written to disk. They are non-ASCII. So while I think in 2011 your command didn't receive a gzipped version, in 2015 the same command did. (wget 1.15). – Peter Cordes Mar 25 at 10:19

simple command to get the html page and compressed it or get any file and compressed.

$ wget -qO - <url> | gzip -c > file_name.gz

for more information about the option . use man command .

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.