Health check of web page using curl

Question

I'd like to do a health check of a service by calling a specific url on it. Feels like the simplest solution would be to use cron to do the check every minute or so. In case of errors, cron sends me an email.

I tried using cUrl for this but I can't get it to output messages only on errors. If I try to direct output to /dev/null, it prints out progress report.

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  5559  100  5559    0     0   100k      0 --:--:-- --:--:-- --:--:--  106k

I tried looking through the curl options but I just can't find anything to suit the situation where you want it to be silent on success but make noise on errors.

Is there a way to make curl do what I want or is there some other tool I should be looking at?

Answer 1

What about -sSf? From the man pages:

  -s/--silent
     Silent or quiet mode. Do not show progress meter or error messages.  
     Makes Curl mute.

  -S/--show-error
     When used with -s it makes curl show an error message if it fails.

  -f/--fail
     (HTTP)  Fail silently (no output at all) on server errors. This is mostly
     done to better enable scripts etc to better deal with failed attempts. In
     normal  cases  when a HTTP server fails to deliver a document, it returns
     an HTML document stating so (which often also describes  why  and  more).
     This flag will prevent curl from outputting that and return error 22.

     This method is not fail-safe and there are occasions where non-successful
     response codes will  slip  through,  especially  when  authentication  is
     involved (response codes 401 and 407).

For example:

curl -sSf http://example.org > /dev/null

Answer 2

I think that for the simplest way to check if the site is alive, you could use the following method:

curl -Is http://www.google.com | head -n 1

This will return HTTP/1.1 200 OK. If the return doesn't match your output then call out for help.

Answer 3

you can capture network timing stats from curl. latencies of each phase in a request/response cycle can be useful to determine health.

$ URL=https://example.com
$ curl "$URL" -s -o /dev/null -w \
> "response_code: %{http_code}\n
> dns_time: %{time_namelookup}
> connect_time: %{time_connect}
> pretransfer_time: %{time_pretransfer}
> starttransfer_time: %{time_starttransfer}
> total_time: %{time_total}
> "
response_code: 200

dns_time: 0.029
connect_time: 0.046
pretransfer_time: 0.203
starttransfer_time: 0.212
total_time: 0.212

Answer 4

You need the -s flag (silent), -f flag (fail with exit code on error) and can use the -o flag to redirect output:

curl www.websiteToTest.com -s -f -o /dev/null || echo "Website down." | mail -s "Website is down" admin@thesite.com 

This is just an bad example for a simple cron script. Normally, you want to get only one mail if a website is down.

Answer 5

Curl has very specific exit status codes
Why not just check for those?

#!/bin/bash

##name: site-status.sh

FAIL_CODE=6

check_status(){
    LRED="\033[1;31m" # Light Red
    LGREEN="\033[1;32m" # Light Green
    NC='\033[0m' # No Color


    curl -sf "${1}" > /dev/null

    if [ ! $? = ${FAIL_CODE} ];then
        echo -e "${LGREEN}${1} is online${NC}"
    else
        echo -e "${LRED}${1} is down${NC}"
    fi
}


check_status "${1}"

Usage:

$ site-status.sh example.com

Result:

$ example.com is online

Notes:

This script only checks if the site can be resolved.

This code should get you by if all you care about is the site being up or down.
However, if you make a few changes to the if/else block, you can easily test for other status codes if you wish

Answer 6

Answer:

#!/usr/bin/env bash
set -euETo pipefail

timeout 3s curl -fIsS http://example.org > /dev/null
# and if you have TLS (https), check if it expires within next 20 days:
true | openssl s_client -connect example.org:443 2>/dev/null | openssl x509 -noout -checkend "$((20*24*60*60))"

Explanation:

• timeout 3s will set a 3-second timeout for your request. Answering slower is consedered to be "not healthy"
• curl -f flag will fail early, -S will show errors, -s will suppress normal output, -I will only fetch HTTP headers, not the content. (As always, more details are available on man curl command.)
• openssl -checkend directive checks expiry dates for a certificate. In my example, it is 20 days (specified in seconds).

Answer 7

I was recently asked to come up with something that would act more like a sophisticated heartbeat.

for i in `curl -s -L cnn.com |egrep --only-matching "http(s?):\/\/[^ \"\(\)\<\>]*" | uniq` ; do curl -s -I $i 2>/dev/null |head -n 1 | cut -d$' ' -f2; done

Or, expanded for a little more readability,

for i in $(curl -s -L cnn.com |egrep --only-matching 'http(s?):\/\/[^ \"\(\)\<\>]*' | uniq)
do
    curl -s -I "$i" 2>/dev/null | head -n 1 | cut -d' ' -f2
done

What I did was curl a website, parse out all the links from the html, and then curl those parsed links, outputting only the status code. Then I would search for http status codes >= 400 to find errors.

Answer 8

This way will help you when trying to test the site when https is present:

#!/bin/bash
# put your domain in this var
https=https://www.somewebsite.com

# save the status in some variable 
status=`curl $https -k -s -f -o /dev/null && echo "SUCCESS" || echo "ERROR"`    

# print results (or use it in your scripts)
echo "testing $https=$status"