2

I wrote a script that shut down all servers in our environment. But if some server is not reachable to begin with or if it is already down, my script just hangs and nothing happens. How may I make it move ahead if some server is unreachable?

#!/bin/bash
#script for Shutting down all VM & BM.
Region=$1
user=$2
region_file_path="/region/$Region.txt"
host=`cat $region_file_path`
key_path="/root/.ssh/id_rsa_adminpod"
for i in $host
do
#       echo "Shutting down Host in $Region with ip addrss $i"
        ssh -i $key_path -p 2222 $user@$i "sudo init 0" &> /dev/null
        if [ $? -ne  0 ]; then
           echo "$i is shutdown!"
        else
           echo "There is some issue, try again"
           exit 1
        fi
done
  • It hangs because it waits for your ssh to timeout, it would probably run through if you wait long enough or add a connecttimeout. – Ziazis Sep 26 '17 at 11:47
5

The simplest solution is to set connection timeout to some reasonable amount of time.

ssh -o ConnectTimeout=10 -i $key_path -p 2222 $user@$i "sudo init 0" &> /dev/null

10 seconds should be enough.

You could also use some other tools for automating tasks - i.e. Ansible.

  • U r right i could try this,, but what if i have dependency on some other task,, which is not responding,, in that scenario how can we make our script go ahead??? – Mohammed Ali Sep 26 '17 at 12:08
  • It's only connection timeout. When timeout is reached ssh returns non zero (255) status, so your scripts can go ahead. example: [~]─> ssh -o ConnectTimeout=1 admin@9gag.com ssh: connect to host 9gag.com port 22: Connection timed out [~]─(255)-> Connection timeout won't check if your command is hanging out. Ex. ssh -o ConnectTimeout=10 root@some.host "sleep 10000" will sleep for 10000 seconds. – Alex Baranowski Sep 26 '17 at 12:25
  • Buddy i got u!! Yes and this would work no doubt!! Actualy now im asking like suppose if we encounter a similiar situation in which our script need to carry on like say #yum install httpd,, and if the internet connection is extremely poor and our script got stuck what else could we use to make our script move ahead – Mohammed Ali Sep 26 '17 at 12:29
  • It's more complicated. The simplest solution is to put command in background in detached state, so in case when ssh session dies the command won't. Once more is the simplest solution, not the best one! You can make it with ssh user@your_host 'nohup Here goes command &'. It will write the command output to nohup.log on your_host machine. But it's not the best solution. The best solution is provided in Ansible/Chief/Puppet/Salt its called idempotence :). – Alex Baranowski Sep 26 '17 at 12:39
1

Solution using Ansible is following.

0) Make sure you have enable SSH passwordless access between management node and nodes to be shutdown.

1) prepare simple inventory file with your nodes which should be shutdown. There is example content:

[local]
localhost ansible_connection=local

[nodes]
192.168.1.30
192.168.1.40

2) Run ansible shell module with your inventory file specified as parameter and shutdown command:

ansible -i /tmp/hosts -m shell -a "/usr/sbin/shutdown +1" nodes

The shutdown has 1 minute delay specified, so the connection isn't killed immediately. But maybe it's not needed.

This is just example, you may use any other command you want to run in parallel on multiple nodes.

0

If u have an 'not responding' dependency you could just go ahead without exit 1, with some more times to retry the ssh connection:

for i in $host
do
        counter=0
        while [ $counter -ne 3 ]; do
        #       echo "Shutting down Host in $Region with ip addrss $i"
                ssh -o ConnectTimeout=10 -i $key_path -p 2222 $user@$i "sudo init 0" &> /dev/null
                if [ $? -ne  0 ]; then
                   echo "$i is shutdown!"
                   counter=3
                else
                   echo "There is some issue, try again"
                   counter=$(($counter+1))
                fi
        done
done
0

A better option would be to first see the server is serving ssh by using NMAP or NC

Nmap: This is very quick, very reliable but requires nmap to be installed Preferred method NMAP (ex host ip 127.0.0.1) :

nmap 127.0.0.1 -PN -p ssh | grep open

Nc: nc is usually installed already , however on some systems such as Mac OS X, the command hangs on unreachable systems. (see workaround)

nc -v -z -w 3 127.0.0.1 22 &> /dev/null && echo "Online" || echo "Offline"

Note: MAC users will have issue with NC not timing out as of writing this post 5/2019

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.