3

I am writing a shell script to download the latest version of some software. After I parse the output of curl I go through several steps to find the exact version string (at the moment of writing this is 0.65.3):

(In all code samples below, > is my prompt. Output from Bash 3.2 or Zsh is on a line without a > prefix.)

> url="https://github.com/gohugoio/hugo/releases/latest"
> latest=$(curl --silent --head "$url" | grep Location)
> tag=$(echo "$latest" | cut -d'/' -f8)
> version=$(echo "${tag//v}")
> echo "hugo_${version}_Linux-64bit.tar.gz"
_Linux-64bit.tar.gz 

The output I expected was hugo_0.65.3_Linux-64bit.tar.gz, but in the output of the call to echo with the quoted string it seems the bytes following ${version} have been used to overwrite the bytes at the beginning of the quoted string.

Here I use two different quoted strings to clarify what is happening:

> echo "hugo_${version}test"
test_0.65.3
> echo "hugo_${version}lorem ipsum dolor sit amet"
lorem ipsum dolor sit amet

I get the same unexpected result if I do this:

> version=$(echo "${tag:1}")
> echo "hugo_${version}_Linux-64bit.tar.gz"
_Linux-64bit.tar.gz 

But, I get the expected result if I do this:

> version=0.65.3
hugo_0.65.3_Linux-64bit.tar.gz

This last result is what is required, but of course it makes my script static rather than dynamic, and thus not very useful to me. How can I get the desired result without hard-coding the value of $version in my script?

  • 2
    Simpler and more robust if GNU sed: version=$(curl -sI $url | grep -i ^Location | sed 's:.*/v::; :s\r$::') HTTP header names are case-insensitive, and I have gotten both Location and location from github. For non-GNU sed can use less-robust :s.$:: . Or use awk: version=$(curl... | awk '/^[Ll]ocation:/{ sub(".*/v",""); sub("\r$",""); print }' ) – dave_thompson_085 Feb 24 at 2:47
  • Thanks for the tip @dave_thompson_085. Indeed I feel that using cut and hard-coding the field number where I expect the version to be is not very robust. – Michiel van Oosterhout Feb 24 at 6:36
16

The line returned by curl ends with carriage-return line-feed. (MS-dos line ending). The line-feed is removed by the Unix tools, however this leaves a carriage-return at the end.

Fix this line to use dos2unix (and quote your argument to echo, avoiding the bugs described in BashPitfalls #14):

version="$(echo "${tag//v}" | dos2unix)"

...or, using the shell's built-in syntax to enact both changes at once:

version=${tag//[$'v\r']/}

dos2unix does make some other changes (like adding a trailing newline after the last line of text, which UNIX requires but DOS does not), but none of them matter for a single-line string like this.

|improve this answer|||||
  • I don't have dos2unix (I'm on macOS), but I'll try this in a Docker container soon. In the mean time, can you expand on your answer? I'd like to understand what exactly is happening. Is it that the line that grep matched from curl's output ends with \r\n? – Michiel van Oosterhout Feb 23 at 21:52
  • 4
    Michiel: yes, HTTP headers are _required to end with the ASCII characters CR LF, notated in many (but not all) programming languages and tools \r \n. – dave_thompson_085 Feb 24 at 2:35
  • 3
    I piped curls output to tr -d '\r' (for lack of dos2unix) before grep and now I get the expected result. Thanks for the explanation – Michiel van Oosterhout Feb 24 at 6:32
14

ctrl-alt-delor’s answer explains why you’re seeing this behaviour; but to address your underlying goal, I recommend using the GitHub API instead of interpreting the “latest” redirection:

version=$(curl https://api.github.com/repos/gohugoio/hugo/releases/latest | jq -r '.tag_name | ltrimstr("v")')

This asks the API for the information on the latest Hugo release, and extracts the tag name using jq, stripping any leading “v”.

Ideally you would even extract the asset names and URLs from the returned JSON.

|improve this answer|||||

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.