How to implement retry mechanism while executing the server url in the shell script?

Question

I am working on a project in which I need to make a url call to one of my server from the bash shell script..

http://hostname.domain.com:8080/beat

After hitting the above url, I will be getting the below response which I need to parse it and extract value of syncs and syncs_behind

state: READY process: 30 process_behind: 100 num_rounds: 60 hour_col: 2 day_col: 0 oldest_day_col: 0

Now I need to hit the above url every 10 seconds for a period of 10 minutes and extract the value of process and process_behind from it and then use it to validate it with below conditions -

process > 8
process_behind = 0

if the syncs is greater than 8 and process_behind = 0, then I will end my shell script with some message saying - "Data has been validated", otherwise I would keep on trying for 10 minute window.. If in that 10 minute window, the above condition doesn't met I will end the shell script anyway meaning I won't retry again.

Below is my shell script which does the above thing and it works fine in normal condition when the server is up.

#!/bin/bash

COUNT=60   #number of 10 second timeouts in 10 minutes

while [[ $COUNT -ge "0" ]]; do

#send the request, put response in variable
DATA=$(wget -O - -q -t 1 http://hostname.domain.com:8080/beat)

#grep $DATA for process and process_behind
PROCESS=$(echo $DATA | grep -oE 'process: [0-9]+' | awk '{print $2}')
PROCESS_BEHIND=$(echo $DATA | grep -oE 'process_behind: [0-9]+' | awk '{print $2}')

echo $PROCESS
echo $PROCESS_BEHIND

#verify conditionals
if [[ $PROCESS -gt "8" && $PROCESS_BEHIND -eq "0" ]]; then exit 0; fi

#decrement the counter
let COUNT-=1

#wait another 10 seconds
sleep 10

done

There might be a scenario in which it gets failed suppose if the server is down, then wget line throws an exception.

Now what I am trying to do is, if the server is down, then I will sleep for 30 seconds and then retry executing the server url again and if it failed again, then sleep again for 30 seconds and then retry executing the server url again. I will retry executing the server url for n times, let's say n is 10.

And after that still the server is not up I will exit out of the shell script with non zero status and a message server is down. But if the server is up and I am able to get the response back, I will move forward to extract those fields which I have in my later shell script.

Is this possible to implement retry mechanism in bash shell script? Or is there any better way of doing this apart from wget?

UPDATE 1:-

This is what I have got -

#!/bin/bash

COUNT=60   #number of 10 second timeouts in 10 minutes
DATA=""
RETRY=10

while [[ $COUNT -ge "0" ]]; do

while [ $RETRY -gt 0 ]
do
    #send the request, put response in variable
    DATA=$(wget -O - -q -t 1 http://machineA:8080/beat)
    echo "Hello"
    if [ $? -eq 0 ]
    then
        break
    else
        let RETRY-=1
        sleep 30
    fi
done

if [ $RETRY -eq 0 ]
then
    exit 2
fi

#grep $DATA for process and process_behind
PROCESS=$(echo $DATA | grep -oE 'process: [0-9]+' | awk '{print $2}')
PROCESS_BEHIND=$(echo $DATA | grep -oE 'process_behind: [0-9]+' | awk '{print $2}')

echo $PROCESS
echo $PROCESS_BEHIND

#verify conditionals
if [[ $PROCESS -gt "8" && $PROCESS_BEHIND -eq "0" ]]; then exit 0; fi

#decrement the counter
let COUNT-=1

#wait another 10 seconds
sleep 10

done

If my server is down and I am running the above shell script, then it is printing "Hello" to the console and it works fine. But see my below update -

Update 2:-

Ok, Now I have found the issue, if I am running the shell script like this which I will be running in my production system and if the server is down, then it doesn't prints out "Hello" at all. But if I run the above shell script in Update 1 and if the server is down, then it works fine.

#!/bin/bash

COUNT=60   #number of 10 second timeouts in 10 minutes
HOSTNAME=machineA
DATA=""
RETRY=10

while [[ $COUNT -ge "0" ]]; do

while [ $RETRY -gt 0 ]
do
    #send the request, put response in variable
    DATA=$(wget -O - -q -t 1 http://$HOSTNAME:8080/beat)
    echo "Hello"
    if [ $? -eq 0 ]
    then
        break
    else
        let RETRY-=1
        sleep 30
    fi
done

if [ $RETRY -eq 0 ]
then
    exit 2
fi

#grep $DATA for process and process_behind
PROCESS=$(echo $DATA | grep -oE 'process: [0-9]+' | awk '{print $2}')
PROCESS_BEHIND=$(echo $DATA | grep -oE 'process_behind: [0-9]+' | awk '{print $2}')

echo $PROCESS
echo $PROCESS_BEHIND

#verify conditionals
if [[ $PROCESS -gt "8" && $PROCESS_BEHIND -eq "0" ]]; then exit 0; fi

#decrement the counter
let COUNT-=1

#wait another 10 seconds
sleep 10

done

This is what I am getting in the debug mode with the above script -

david@some-machine:~$ bash -x ./ping1.sh
+ set -e
+ COUNT=60
+ HOSTNAME=machineA
+ DATA=
+ RETRY=10
+ echo machineA
machineA
+ [[ 60 -ge 0 ]]
+ '[' 10 -gt 0 ']'
++ wget -O - -q -t 1 http://machineA:8080/beat
+ DATA=

I guess both of the scripts are same? Then why this is happening?

Answer 1

You can use a simple loop:

....
DATA=""
RETRY=10

while [ "$RETRY" -gt 0 ]; do
  DATA="$(wget -O - -q -t 1 http://hostname.domain.com:8080/beat)"
  if [ $? -eq 0 ]
  then
    break
  else
    let RETRY-=1
    sleep 30
  fi
done

if [ "$RETRY" -eq 0 ]
then
  exit 2
fi
....