11

Not sure if it is ok to share the website I tried to get its source, but I think it is necessary for a better explanation. And I apologize if it's not in advance

The command: curl -k -L -s https://www.mi.com

The output was binary data for some reason by getting the following error

Warning: Binary output can mess up your terminal. Use "--output -" to tell
Warning: curl to output it to your terminal anyway, or consider "--output
Warning: <FILE>" to save to a file.

How can I read the page HTML source? thanks!

4 Answers 4

27

The returned data is compressed, you can instruct curl to handle the decompression directly by adding the --compressed option:

curl -k -L -s --compressed https://www.mi.com
0
4

Just redirect it to a file, then you can investigate what it is:

curl -k -L -s https://www.mi.com > outFile

You can now use the file command to see what outFile contains:

$ file outFile 
outFile: gzip compressed data, from Unix, original size modulo 2^32 135402

So, you just downloaded compressed data. To see it, either decompress:

mv outFile outFile.gz ## gzip requires the .gz extension
gunzip outFile.gz

Or just use a tool that can handle compressed data such as zmore:

zmore outFile

Or zcat:

zcat outFile
2
  • Wow, didn't know they exist .. can I edit and add links for zcat, zmore ?
    – hc_dev
    Jan 13 at 19:37
  • @hc_dev you can always edit! That said, these are standard utilities, either installed by default or trivially available in the operating system's repositories, so I don't have nt really see why a link would be helpful, but this site is a collaborative effort so if you think they're useful, go ahead!
    – terdon
    Jan 13 at 22:51
2

Use "--output -" to tell Warning: curl to output it to your terminal anyway

like in

curl -k -L --output - s https://www.mi.com

or --output file to save it to a file like in

curl -k -L --output filename s https://www.mi.com

and examine the file with your favourite editor.

1
0

If you don't know what format the binary output has it would be safer to inspect the output format first.

Otherwise as you already use -s, for silent mode like in batch/automation scripts, you could directly pipe it on to a suitable binary-converter.

Or, assuming the HTTP output was compressed, simply decompress it.

Inspecting binary curl output

  1. Use -I to see response-headers only and view Content- headers for a given media type (also known as MIME type) indicating the (binary) format:
curl -k -L -s https://www.mi.com -I

E.g. curl -k -L -s https://www.mi.com -I | grep Content- will also show other output-metadata like encoding (e.g. UTF-8), length in bytes (here 169747 bytes, after redirection):

Content-Type: text/html

Content-Length: 223

Content-Type: text/html; charset=utf-8

Content-Length: 169747

Note: above HTTP request was redirected, thus we get 2 responses and 2 response-headers from curl. The first is from redirection, the second from final output.

  1. Pipe to type to inspect the binary data type (file-type):
curl -k -L -s https://www.mi.com | type
  1. Using -o or --output to save to a file:
curl -k -L -s https://www.mi.com -o outFile

Same like redirection > outFile suggested by terdon.

Passing binary output on (e.g. to an image-viewer)

You can also use the pipe to pass e.g. an image indicated by response-header Content-Type: image/x-icon to a stdin-accepting viewer like feh to show it:

curl -k -L -s https://www.mi.com/favicon.ico | feh

De-compressing curl output

Binary output may be the result of HTTP compression which is often used to save bandwidth and speed-up transmission.

  1. Piping to zcat to view unzipped directly:
curl -k -L -s https://www.mi.com | zcat

If curl's output was not zipped as expected following warning will show:

gzip: stdin: not in gzip format

As terdon explained interactively.

  1. Using --compressed:
curl -k -L -s https://www.mi.com --compressed

As Stephen already answered.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.