Why do i get binary output using curl

Question

Not sure if it is ok to share the website I tried to get its source, but I think it is necessary for a better explanation. And I apologize if it's not in advance

The command: curl -k -L -s https://www.mi.com

The output was binary data for some reason by getting the following error

Warning: Binary output can mess up your terminal. Use "--output -" to tell
Warning: curl to output it to your terminal anyway, or consider "--output
Warning: <FILE>" to save to a file.

How can I read the page HTML source? thanks!

Answer 1

The returned data is compressed, you can instruct curl to handle the decompression directly by adding the --compressed option:

curl -k -L -s --compressed https://www.mi.com

Answer 2

Just redirect it to a file, then you can investigate what it is:

curl -k -L -s https://www.mi.com > outFile

You can now use the file command to see what outFile contains:

$ file outFile 
outFile: gzip compressed data, from Unix, original size modulo 2^32 135402

So, you just downloaded compressed data. To see it, either decompress:

mv outFile outFile.gz ## gzip requires the .gz extension
gunzip outFile.gz

Or just use a tool that can handle compressed data such as zmore:

zmore outFile

Or zcat:

zcat outFile

Answer 3

Use "--output -" to tell Warning: curl to output it to your terminal anyway

like in

curl -k -L --output - s https://www.mi.com

or --output file to save it to a file like in

curl -k -L --output filename s https://www.mi.com

and examine the file with your favourite editor.

Answer 4

If you don't know what format the binary output has it would be safer to inspect the output format first.

Otherwise as you already use -s, for silent mode like in batch/automation scripts, you could directly pipe it on to a suitable binary-converter.

Or, assuming the HTTP output was compressed, simply decompress it.

Inspecting binary curl output

1. Use -I to see response-headers only and view Content- headers for a given media type (also known as MIME type) indicating the (binary) format:

curl -k -L -s https://www.mi.com -I

E.g. curl -k -L -s https://www.mi.com -I | grep Content- will also show other output-metadata like encoding (e.g. UTF-8), length in bytes (here 169747 bytes, after redirection):

Content-Type: text/html

Content-Length: 223

Content-Type: text/html; charset=utf-8

Content-Length: 169747

Note: above HTTP request was redirected, thus we get 2 responses and 2 response-headers from curl. The first is from redirection, the second from final output.

2. Pipe to type to inspect the binary data type (file-type):

curl -k -L -s https://www.mi.com | type

3. Using -o or --output to save to a file:

curl -k -L -s https://www.mi.com -o outFile

Same like redirection > outFile suggested by terdon.

Passing binary output on (e.g. to an image-viewer)

You can also use the pipe to pass e.g. an image indicated by response-header Content-Type: image/x-icon to a stdin-accepting viewer like feh to show it:

curl -k -L -s https://www.mi.com/favicon.ico | feh

De-compressing curl output

Binary output may be the result of HTTP compression which is often used to save bandwidth and speed-up transmission.

1. Piping to zcat to view unzipped directly:

curl -k -L -s https://www.mi.com | zcat

If curl's output was not zipped as expected following warning will show:

gzip: stdin: not in gzip format

As terdon explained interactively.

2. Using --compressed:

curl -k -L -s https://www.mi.com --compressed

As Stephen already answered.