ssh into host and running service check script

Question

I need to ssh and perform service check on multiple hosts

say List of hosts: host1 host2 host3

I wrote the script as follows but it is getting stuck at ssh only

#!/bin/bash

service=splunk

for host in $(</tmp/tstuf.txt);
do
  ssh $host;
  if (( $(ps -ef | grep -v grep | grep $service | wc -l) > 0 ))
  then
    echo "$service is running"
  else
    echo "$service is not running"
  fi
done

while running the script it is running ssh and going on to host1 from current host and stuck there.. It is not running service check. Can someone please suggest

Answer 1

Your script is "getting stuck at ssh only" because you are not giving it a command to execute. The ssh $host just establishes an interactive ssh session with $host, which then waits for you to start interacting with it.

When you exit from the remote host, your script will continue and run if (( $(ps ....) > 0 )); then .......... fi on your LOCAL system. And then continue the for loop with the next host in the list (getting "stuck" again until you exit).

#!/bin/bash

service=splunk

for host in $(</tmp/tstuf.txt); do
  ssh $host "if (( \$(ps -ef | grep -v grep | grep $service | wc -l) > 0 )); then
               echo $service is running
             else
               echo $service is not running
             fi"
done

or, as @DopeGhoti suggested, a much simpler, better alternative is:

#!/bin/sh

service=splunk

for host in $(</tmp/tstuf.txt); do
  # echo -n "$host:"  # optional. uncomment the echo if wanted.
  ssh "$host" service "$service" status
done

---

An even better alternative is to install something like LLNL's Parallel Distributed Shell aka pdsh, then (after configuring the hosts list file in /etc/genders) you can just run commands like:

pdsh -g all service splunk status

that runs service splunk status on the group of hosts defined by the "all" tag.

pdsh runs the commands on multiple hosts at the same time (hence the name "parallel distributed shell"), rather than one at a time. Each line of output from the remote system is prefixed by its hostname.

e.g. on my own small home network:

$ pdsh -g all uptime
kali:  12:09:03 up 3 days, 23:26,  2 users,  load average: 0.08, 0.05, 0.06
hanuman:  12:09:03 up 12 days, 13:09,  2 users,  load average: 0.00, 0.00, 0.00
indra:  12:09:03 up 12 days, 13:05,  2 users,  load average: 0.07, 0.12, 0.09
ganesh:  12:09:03 up 34 days, 23:34, 19 users,  load average: 1.86, 1.48, 1.40

BTW, single-line output like the uptime example above can be formatted into neat columns by piping into, e.g., column -t. multi-line output can be grouped by hostname by piping into pdsh's companion utility dshbak.

If you don't want to be entering passwords all the time, it requires ssh key-based authentication to be set up on each remote host...as would your for loop or anything else that connects via ssh.

pdsh is packaged for debian, and most other linux distributions.

Answer 2

Inner loop in a single line. the return status of 'pgrep' is returned by ssh to the local host.

echo "$host: $service is $(ssh $host pgrep $service >/dev/null || echo "NOT ")running"