4
#!/bin/bash
echo "$(curl -s -I https://google.com|grep Server)" abc

returns:

abcer: gws

and bash -x shows:

++ grep Server
++ curl -s -I https://google.com
' abco 'Server: gws
 abcer: gws

but

echo abc "$(curl -s -I https://google.com|grep Server)"` 

returns

abc Server: gws

In other words, if I add text AFTER the curl command, it gets mangled, but it's ok if I add it before curl.
What is happening? I cannot figure it out.

3 Answers 3

5

The HTTP standard specifies that all header lines, as well as the empty line that marks the end of the headers, must use CRLF (carriage return, line feed) endings. A lot of clients are liberal and accept LF alone, but most servers, including Google, respect the standard.

curl -I displays the headers exactly as sent by the server, including the CR characters. As far as a Unix system is concerned, a CR character is part of the line, it's only the LF character that ends the line. Command substitution removes trailing LF characters but doesn't touch CR characters. So, the command substitution $(curl -s -I https://google.com|grep Server) returns (in your case) the string Server: gws␍ (where is a carriage return), and the command echo "$(curl -s -I https://google.com|grep Server)" abc displays a line containing

Server: gws␍abc

The character is an instruction to the terminal to move the cursor to the beginning of the line, so the characters abc overwrite the first three characters of the line.

This also explains why the trace from bash -x is somewhat garbled.

The command echo abc "$(curl -s -I https://google.com|grep Server)" displays a line containing

Server: gwsabc␍

The ␍ character moves the cursor to the beginning of the line, then the linefeed moves the cursor to the beginning of the next line, so the ␍ has no visible effect.

To avoid complications due to CR, strip them from the input.

echo "$(curl -s -I https://google.com | tr -d '\r' | grep Server)" abc
2
echo $(curl -s -I https://google.com|grep Server)|cat -A

shows that the value returned by curl has a ^M (a carriage return). When you print out the output of the curl, the carriage return takes the 'cursor' back to the start of the line, where it then prints out the ' abc', overwriting 'Serv'.

In your second attempt, the carriage return has no obvious effect.

2

Try this to remove carriage return:

 echo "$(curl -s -I https://google.com|grep Server)" abc | tr -d "\r"

Output:

Server: GFE/2.0 abc

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.