Automatically run commands over SSH on many servers

Question

There is a list of IP addresses in a .txt file, ex.:

1.1.1.1
2.2.2.2
3.3.3.3

Behind every IP address there is a server, and on every server there is an sshd running on port 22. Not every server is in the known_hosts list (on my PC, Ubuntu 10.04 LTS/bash).

How can I run commands on these servers, and collect the output?

Ideally, I'd like to run the commands in parallel on all the servers.

I'll be using public key authentication on all the servers.

Here are some potential pitfalls:

• The ssh prompts me to put the given servers ssh key to my known_hosts file.
• The given commands might return a nonzero exit code, indicating that the output is potentially invalid. I need to recognize that.
• A connection might fail to be established to a given server, for example because of a network error.
• There should be a timeout, in case the command runs for longer than expected or the server goes down while running the command.

The servers are AIX/ksh (but I think that doesn't really matter.

Answer 1

There are several tools out there that allow you to log in to and execute series of commands on multiple machines at the same time. Here are a couple:

• pssh
• pdsh
• clusterssh
• clusterit
• mussh

Answer 2

If you're into Python scripting more than bash scripting, then Fabric might be the tool for you.

From the Fabric home page:

Fabric is a Python (2.5 or higher) library and command-line tool for streamlining the use of SSH for application deployment or systems administration tasks.

It provides a basic suite of operations for executing local or remote shell commands (normally or via sudo) and uploading/downloading files, as well as auxiliary functionality such as prompting the running user for input, or aborting execution.

Typical use involves creating a Python module containing one or more functions, then executing them via the fab command-line tool.

Answer 3

Assuming that you are not able to get pssh or others installed, you could do something similar to:

tmpdir=${TMPDIR:-/tmp}/pssh.$$
mkdir -p $tmpdir
count=0
while IFS= read -r userhost; do
    ssh -n -o BatchMode=yes ${userhost} 'uname -a' > ${tmpdir}/${userhost} 2>&1 &
    count=`expr $count + 1`
done < userhost.lst
while [ $count -gt 0 ]; do
    wait $pids
    count=`expr $count - 1`
done
echo "Output for hosts are in $tmpdir"

Answer 4

I do use GNU parallel for that, most specifically you can use this recipe:

parallel --tag --nonall --slf your.txt command

With your.txt being the file with the server IP address/names.

Answer 5

Very basic setup:

for host in $(cat hosts.txt); do ssh "$host" "$command" >"output.$host"; done

Authenticating with name/password is really no good idea. You should set up a private key for this:

ssh-keygen && for host in $(cat hosts.txt); do ssh-copy-id $host; done

Answer 6

I suggest Ansible.cc. It's a configuration manager and command dispatcher.

Answer 7

I think you're looking for pssh and the related parallel versions of the usual scp, rsync, etc..

Answer 8

I built an open-source tool called Overcast to make this sort of thing easier.

First you define your servers:

overcast import vm.01 --ip 1.1.1.1 --ssh-key /path/to/key
overcast import vm.02 --ip 2.2.2.2 --ssh-key /path/to/key

Then you can run multiple commands and script files across them, either sequentially or in parallel, like so:

overcast run vm.* uptime "free -m" /path/to/script --parallel

Answer 9

The Hypertable project has recently added a multi-host ssh tool. This tool is built with libssh and establishes connections and issues commands asynchronously and in parallel for maximum parallelism. See Multi-Host SSH Tool for complete documentation. To run a command on a set of hosts, you would run it as follows:

$ ht ssh 1.1.1.1,2.2.2.2,3.3.3.3 uptime

You can also specify a host name or IP pattern, for example:

$ ht ssh 1.1.1.[1-99] uptime
$ ht ssh host[00-99] uptime

It also supports a --random-start-delay <millis> option that will delay the start of the command on each host by a random time interval between 0 and <millis> milliseconds. This option can be used to avoid thundering herd problems when the command being run accesses a central resource.

Answer 10

Just a headsup for a really nice question:

Best solution I've found is, not-so-surprisingly, tmux.

You can do Ctrl-B : setw synchronize-panes in tmux for amazing resut. You must have all of your ssh connections opened, in different panes of 1 window of Tmux for this.

Answer 11

Maybe something like this works, to run command on multiple hosts? suppose all hosts are setup in .ssh/config for password-less login.

$ < hosts.txt xargs -I {} ssh {} command

Answer 12

Create a file /etc/sxx/hosts

populate like so:

[grp_ips]
1.1.1.1
2.2.2.2
3.3.3.3

share ssh key on all machines.

Install sxx from package:

https://github.com/ericcurtin/sxx/releases

Then run command like so:

sxx username@grp_ips "whatever bash command"

Answer 13

Use Paramiko http://www.paramiko.org/ and Thread-based parallelism https://docs.python.org/3/library/threading.html .below code allows to execute multiple commands on each server in parallel!

ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.load_system_host_keys()
ssh.connect(hostname, port, username, password)
t = threading.Thread(target=tasks[index], args=(ssh, box_ip,))
t.start()

Note: repeat above instructions for each server.

Answer 14

I think "expect" is what you need.
Many people does not even know it exists. It's a tcl based program that will made question and possible answers on your behalf. Have a look to https://wiki.tcl-lang.org/page/Expect

Answer 15

Use pdsh. Here's a useful example:

pdsh -w 'something.example.com,node-[1-5].example.com' -R ssh 'uptime'