3

File req contains the request header:

GET /cd/E11882_01/server.112/e41084/toc.htm HTTP/1.1^M
Host: docs.oracle.com^M
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8^M
Accept-Language: en-US,en;q=0.5^M
Accept-Encoding: gzip, deflate^M
Connection: keep-alive^M
^M

I run:

cat req | nc docs.oracle.com 80 > resp

resp contains:

HTTP/1.1 200 OK^M
Server: Apache^M
ETag: "726bf43b293f9fc8eac0f8f6b7be3a84:1457459134"^M
Last-Modified: Fri, 04 Mar 2016 14:26:34 GMT^M
Accept-Ranges: bytes^M
Content-Type: text/html^M
Vary: Accept-Encoding^M
Content-Encoding: gzip^M
Date: Sat, 18 Jun 2016 07:04:06 GMT^M
Content-Length: 13163^M
Connection: keep-alive^M
^M
^_<8b>^H^@^@^@^@^@^@^@Å}ysã8<92>ïÿó)¸Þ<88>}3ïµËâMÎvy<83>â%ªuµ(Õ1^[^[
Z¢mvÉ<92>[Gu¹?ýf<82>^D^Hɦ  Òîx^[³]¶¬ü^AH$^R<99><89>Dâç^?óÆîìëÄ<97>î^O^Oëë¿ý<8c>ÿHëds÷ñ"Ý\à^Gi²<82>^?^^ÒC^Bß9<^¦¿^_³ï^_/¾\Î<9d>Kwûð<98>^\²<9b>uz!-·<9b>Cº9|¼<88>ü<8f>éê.½ T<9b>ä!ýxñ=KÿxÜî^NÜ^WÿÈV<87>û<8f>«ô{¶L/É/?IÙ&;dÉúr¿LÖéGùCç'é!ù<91>=^\^_èG^Lwy<9f>ìö)à^\^O·<97>^V~|È^NëôÚK^NÉM²O¥ø×<81>4<80>¡^\<93>»T<9a>¦·é.Ý,SéRró^^ì^?¾Ê)N:z<97>nÒ]rØî¸<9e><8e>wÉr<9d>J<9e>3íJ_z³á^@!¾§»Cº<93>þ>Ü®R飴Ú.<8f>^Oðí^?@^CÃtw<97>®¤Oén<9f>m7<92>Ü1õ^Kéê´<9d>Õ^R¨^_ö^_<96>»49¤+®5¥#^[<97>^]ù²£Ïô^?jÆ?^Uë_ϨwÛ<9b>íaÏ^Q%ëue^Sd<94>Üwk8T<89><93>­<80><»ÍR<9e>7¾&w,í²£U<93>í^KF<8c>o9:h{^Zä4ëlóMÚ¥køð<90> <88>ÜïÒÛ<8f>^W^_>\Áÿ²Í*ýñ^AäòB"ãøxÑÛ>@^_^OO<8f>ðó!ýq¸B¡=Gr·<8f>O»ìîþ^LmµÜ><l7<84>äj    _9Aæ<88>^<82>ÿÛÏûå.{<^T^?L^^^_×Ù^Rä^_ð~K¾'ù^_/$i¿[<9e>·÷Ûþ

   ...continues...

Now, apparently the response body is in gzip format. To decompress it, I have copied the response body to resp-body. So, resp-body contains:

^_<8b>^H^@^@^@^@^@^@^@Å}ysã8<92>ïÿó)¸Þ<88>}3ïµËâMÎvy<83>â%ªuµ(Õ1^[^[
Z¢mvÉ<92>[Gu¹?ýf<82>^D^Hɦ  Òîx^[³]¶¬ü^AH$^R<99><89>Dâç^?óÆîìëÄ<97>î^O^Oëë¿ý<8c>ÿHëds÷ñ"Ý\à^Gi²<82>^?^^ÒC^Bß9<^¦¿^_³ï^_/¾\Î<9d>Kwûð<98>^\²<9b>uz!-·<9b>Cº9|¼<88>ü<8f>éê.½ T<9b>ä!ýxñ=KÿxÜî^NÜ^WÿÈV<87>û<8f>«ô{¶L/É/?IÙ&;dÉúr¿LÖéGùCç'é!ù<91>=^\^_èG^Lwy<9f>ìö)à^\^O·<97>^V~|È^NëôÚK^NÉM²O¥ø×<81>4<80>¡^\<93>»T<9a>¦·é.Ý,SéRró^^ì^?¾Ê)N:z<97>nÒ]rØî¸<9e><8e>wÉr<9d>J<9e>3íJ_z³á^@!¾§»Cº<93>þ>Ü®R飴Ú.<8f>^Oðí^?@^CÃtw<97>®¤Oén<9f>m7<92>Ü1õ^Kéê´<9d>Õ^R¨^_ö^_<96>»49¤+®5¥#^[<97>^]ù²£Ïô^?jÆ?^Uë_ϨwÛ<9b>íaÏ^Q%ëue^Sd<94>Üwk8T<89><93>­<80><»ÍR<9e>7¾&w,í²£U<93>í^KF<8c>o9:h{^Zä4ëlóMÚ¥køð<90> <88>ÜïÒÛ<8f>^W^_>\Áÿ²Í*ýñ^AäòB"ãøxÑÛ>@^_^OO<8f>ðó!ýq¸B¡=Gr·<8f>O»ìîþ^LmµÜ><l7<84>äj    _9Aæ<88>^<82>ÿÛÏûå.{<^T^?L^^^_×Ù^Rä^_ð~K¾'ù^_/$i¿[<9e>·÷Ûþ

   ...continues...

Then I have tried gzip -d resp-body but it does not work.

What should I do in order to decompress the response?

6

Delete the headers and what you'll have left is gzip-compressed data that can be decompressed with gzip -d or zcat. e.g.

sed -e '1,/^[[:space:]]*$/d' resp | gzip -d > resp.decompressed

The sed script deletes the headers - i.e. everything from the first line to the first empty line (/^[[:space:]]*$/).

The [[:space:]] character-class will make the sed script match empty lines and lines containing only space characters (including carriage-returns, ^M)

BTW, a slightly smarter version of this would extract the Content-Encoding: and Content-Type: headers, and use the mime-type from that to decide whether to use cat, lynx -dump, gzip -d, bzip2 -d, xz -d or whatever else to "decode" the data. But that would probably require writing it in perl.

  • Wow it worked. But to me, it seems like this method and what I did are the same, yet my method does not work. Could you please explain how is this different than my method (and what is missing in my method)? (I have edited the question in case my method was not clear) – Utku Jun 18 '16 at 8:29
  • 1
    Maybe something changes when I copy the response body with ^C to clipboard and than paste it to resp-body with ^V? – Utku Jun 18 '16 at 8:36
  • it looks like the response has been opened in less which reinterprets control characters into printable ASCII, the result will not after cut-and paste resemble the original, – Jasen Jun 18 '16 at 9:41
  • 1
    @Jasen Actually I opened it in vim but still a similar issue I guess? – Utku Jun 18 '16 at 9:47
  • yeah, absolutely. – Jasen Jun 18 '16 at 9:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.