Resolve filename from a remote URL without downloading a file

Question

I am creating a script, which should download latest version of an application from repository and deploy the app.

The main issue: there are several repositories and I need to check, which of them has most recent version.

E.g.

http://repo1/xyz/LATEST -> (redirects to) -> http://repo1/xyz/app-1.0.0.0.zip
http://repo2/xyz/LATEST -> (redirects to) -> http://repo1/xyz/app-1.1.0.0.zip

So I need to iterate over available repositories and get only a filename - no need to download obsolette versions of software.

Answer 1

Something like this could work for you:

curl -sIkL http://repo1/xyz/LATEST | sed -r '/filename=/!d;s/.*filename=(.*)$/\1/'

Take a look at man page curl(1) for the options. The interesting one is -I, --head.

Explanation as requested per comments:

The idea is to request the HTTP response header only.

Therefore the -I options is used. -s silents curl to not print anything else than the header. -k allows "insecure" SSL connections (curl would reject self-signed certs otherwise). And -L to follow HTTP(S) location redirects.

Then sed(1) is used to get the file name from response header. We are searching for the filename= field, so the /filename=/!d part removes anything without that field from output. Finally the s/.*filename=(.*)$/\1/ part prints the file name only if the field is found.

Answer 2

I came out with this solution, quite similar to the one by @FloHimself:

curl -L --head http://repo1/xyz/LATEST 2>/dev/null | grep Location: | tail -n1 | cut -d' ' -f2

• -L lets curl follow the redirection.
• --head makes it fetch only the headers and not the pages' content.
• grep Location: looks for Location: header in 30x HTTP responses by the server
• tail -n1 selects the last one
• cut -d' ' -f2 selects the second field (the URL)

The same, but letting curl do all the work:

curl -L --head -w '%{url_effective}' http://repo1/xyz/LATEST  2>/dev/null | tail -n1

This solution uses the -w, --write-out option to ask curl for a specific output. man curl gives the available variables.

Answer 3

The answer @lgeorget provided is very close, but it is still not ideal. The better solution is to use filename_effective variable for the --write-out. This is still not completely ideal because it does not work without downloading the file. It is also mentioned in the man curl:

filename_effective
   The ultimate filename that curl  writes  out  to.
   This  is only meaningful if curl is told to write
   to a file with the --remote-name or --output  op‐
   tion.  It's  most  useful in combination with the
   --remote-header-name option.

but if you already have the file in the current folder, then it will throw an error and will discard the downloaded file (in interactive mode it will ask you if you want to overwrite, but in non-interactive mode i.e in a script it throws error):

curl --remote-name \
     --remote-header-name \
     --location \
     --write-out '%{filename_effective}' \
     "${URL}"

Therefore, if you want to use it in a script you can do something like:

# just an example URL which will download Bitwarden's CLI software
URL='https://vault.bitwarden.com/download/?app=cli&platform=linux'

TMP_DOWNLOAD_RESULT="$(curl --remote-name \
                            --remote-header-name \
                            --location \
                            --write-out '%{exitcode};%{errormsg};%{filename_effective}' \
                            "${URL}")"

## break the result into its sub-parts
TMP_EXIT_CODE="$(echo "${TMP_DOWNLOAD_RESULT}" | cut -d ';' -f 1)"
TMP_EXIT_MESSAGE="$(echo "${TMP_DOWNLOAD_RESULT}" | cut -d ';' -f 2)"
TMP_REMOTE_FILE_NAME="$(echo "${TMP_DOWNLOAD_RESULT}" | cut -d ';' -f 3)"

# generate error or message of curl has failed.
if [ "${TMP_EXIT_CODE}" != "0" ]; then
    echo
    if [ "${TMP_EXIT_CODE}" == "23" ]; then
        echo "The local version is already the latest version."
        exit 0
    else
        echo "Downloading with cURL resulted in error number ${TMP_EXIT_CODE} with the message: \n\t${TMP_EXIT_MESSAGE}"
        exit 1
    fi
fi