# Automatically run commands over SSH on many servers

There is a list of IP addresses in a .txt file, ex.:

1.1.1.1
2.2.2.2
3.3.3.3


Behind every IP address there is a server, and on every server there is an sshd running on port 22. Not every server is in the known_hosts list (on my PC, Ubuntu 10.04 LTS/bash).

How can I run commands on these servers, and collect the output?

Ideally, I'd like to run the commands in parallel on all the servers.

I'll be using public key authentication on all the servers.

Here are some potential pitfalls:

• The ssh prompts me to put the given servers ssh key to my known_hosts file.
• The given commands might return a nonzero exit code, indicating that the output is potentially invalid. I need to recognize that.
• A connection might fail to be established to a given server, for example because of a network error.
• There should be a timeout, in case the command runs for longer than expected or the server goes down while running the command.

The servers are AIX/ksh (but I think that doesn't really matter.

-
possible duplicate of How can I reproduce commands run on one machine on another machine? –  Caleb Aug 19 '11 at 12:51
I thinks it's not a duplicate, because the link you mention doesn't even contain SSH. –  LanceBaynes Aug 19 '11 at 13:06
If you have not done so, you should set up ssh servers on all of the machines, make a private/public key pair on the machine you work from and copy the public key to the accounts on the server to prevent further password hassles. That applies for my answer, and for @demure's as well. –  Anthon May 23 '13 at 4:49

Assuming that you are not able to get pssh or others installed, you could do something similar to:

tmpdir=${TMPDIR:-/tmp}/pssh.$$count=0 while userhost; do ssh -n -o BatchMode=yes${userhost} 'uname -a' > ${tmpdir}/${userhost} 2>&1 &
count=expr $count + 1 done < userhost.lst while [$count -gt 0 ]; do
wait $pids count=expr$count - 1
done
echo "Output for hosts are in $tmpdir"  - what does exactly wait do in this script?? ty! – LanceBaynes Aug 19 '11 at 15:43 what happens if a servers key is not in my knows_hosts file? – LanceBaynes Aug 19 '11 at 15:50 putting scripts into the background creates child processes. When a child process exits, the process 'slot' and resources stay in the system until the parent process exits or the parent process 'waits' for the child. These 'terminated by still present' processes are called 'zombie' processes. It is good behavior to clean up after child proceses and reclaim resources. – Arcege Aug 19 '11 at 19:38 If not known, then that might screw it up, but you can add -o StrictHostKeyChecking=no to avoid this. However it is better to scan all the servers from a command-line first so the host keys can be added. – Arcege Aug 19 '11 at 19:40 As I mentioned, after the ssh program is put into the background and after it exits, resources are still kept. The wait command reclaims those resources, including the process' return code (how the processs exited). Then if later in the program, you put another process, lets say a compression, in the background and needs to wait for it, then without this loop, the wait will reclaim one of the completed ssh programs, not the compression - which may still be running. You would get an return status on the wrong child process. It is good to clean up after yourself. – Arcege Aug 28 '11 at 18:23 There are several tools out there that allow you to log in to and execute series of commands on multiple machines at the same time. Here are a couple: - You might want to add pdsh to the list. – Riccardo Murri Aug 19 '11 at 13:17 If you're into Python scripting more than bash scripting, then Fabric might be the tool for you. From the Fabric home page: Fabric is a Python (2.5 or higher) library and command-line tool for streamlining the use of SSH for application deployment or systems administration tasks. It provides a basic suite of operations for executing local or remote shell commands (normally or via sudo) and uploading/downloading files, as well as auxiliary functionality such as prompting the running user for input, or aborting execution. Typical use involves creating a Python module containing one or more functions, then executing them via the fab command-line tool. - I do use GNU parallel for that, most specifically you can use this recipe: parallel --tag --nonall --slf your.txt command  With your.txt being the file with the server IP address/names. - isn't there any other way only using ssh Since I am using my company servers I don't want to install additional packages – Özzesh May 23 '13 at 4:49 Sure, I used to do this before ssh existed as well, but you need some way to login to the other computers and those ways used to be less secure (like rsh). You don't describe what you are using now to get from machine to machine (telnet?, rsh?). – Anthon May 23 '13 at 4:52 @Özzesh yes only on the controller – Anthon May 23 '13 at 5:20 @Özzesh See if your reason for not installing GNU Parallel is covered on oletange.blogspot.dk/2013/04/why-not-install-gnu-parallel.html – Ole Tange May 23 '13 at 8:07 @OleTange Just realised who commented on my answer %-). Thanks for this excellent piece of software. – Anthon May 23 '13 at 8:18 Very basic setup: for host in$(cat hosts.txt); do ssh "$host" "$command" >"output.$host"; done  Authenticating with name/password is really no good idea. You should set up a private key for this: ssh-keygen && for host in$(cat hosts.txt); do ssh-copy-id $host; done  - this is the one I have been looking for !!!!! Thanks michas – Özzesh May 23 '13 at 8:59 the above script works well for me but after executing in 10 servers the script doesnot responds. How can I logout form ssh server that I am connected to ? – Özzesh May 27 '13 at 4:39 You should not need to logout explicitly. This "script" will just run ssh$host $command for each host. Try to run those commands manually to find out what is going on. – michas May 27 '13 at 11:19 The Hypertable project has recently added a multi-host ssh tool. This tool is built with libssh and establishes connections and issues commands asynchronously and in parallel for maximum parallelism. See Multi-Host SSH Tool for complete documentation. To run a command on a set of hosts, you would run it as follows: $ ht ssh 1.1.1.1,2.2.2.2,3.3.3.3 uptime


You can also specify a host name or IP pattern, for example:

$ht ssh 1.1.1.[1-99] uptime$ ht ssh host[00-99] uptime


It also supports a --random-start-delay <millis> option that will delay the start of the command on each host by a random time interval between 0 and <millis> milliseconds. This option can be used to avoid thundering herd problems when the command being run accesses a central resource.

-

I think you're looking for pssh and the related parallel versions of the usual scp, rsync, etc..

-
While this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference. Link-only answers can become invalid if the linked page changes. –  slm Jun 18 '14 at 11:07
This is one vote away from being deleted, but this later answer is at +10. Makes perfect sense –  Michael Mrozek Jun 19 '14 at 17:52
@MichaelMrozek It's worse than you know. The second of the two VTD's is mine on a click-happy-accident. I've had this tab open for a couple days intending to flag or ping you to revert my VTD if it got deleted. Maybe you could just flip it now to clear the votes. –  Caleb Jun 20 '14 at 16:53
@Caleb Fixed it –  Michael Mrozek Jun 20 '14 at 17:32

I suggest Ansible.cc. It's a configuration manager and command dispatcher.

-

I built an open-source tool called Overcast to make this sort of thing easier.

overcast import vm.01 --ip 1.1.1.1 --ssh-key /path/to/key

overcast run vm.* uptime "free -m" /path/to/script --parallel