1

This question already has an answer here:

I have 1000s of remote machines. I want to run some specific commands on all of them, but in parallel. I use these commands which run sequentially:

for f in `cat host.lst`
do
       ./runScript.sh $f
done

Let's suppose host.lst contains 100 hosts. I want to run runScript.sh on 100 hosts in parallel. Also, logs should be maintained.

I can not install any utility on my machine such as PSSH.

I have done a lot of research and found these links but they did not help. I do not understand how they work:

Automatically run commands over SSH on many servers

Execute command on multiple files matching a pattern in parallel

Can any one explain the logic?

marked as duplicate by Anthony Geoghegan, Archemar, Jeff Schaller, GAD3R, Kiwy Apr 12 '17 at 12:22

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • 1
    There are answers in both the questions that you referenced that don't require pssh. Some use GNU parallel (which would be my choice) but there are others that use only commands that are commonly installed on Unix-like systems. I'd advise that you edit the question to show which of those answers you've tried and detail the problems you encountered. As it stands, this question is a duplicate. – Anthony Geoghegan Apr 12 '17 at 8:55
  • See if you understand gnu.org/software/parallel/… You do not need to be root to use GNU Parallel - even if it is not installed: oletange.blogspot.dk/2013/04/why-not-install-gnu-parallel.html – Ole Tange May 4 '17 at 22:35
6
logdir=`mktemp -d`
bunch=200

IFS=$'\n'
for hosts in $(< hosts.lst xargs -r -L "$bunch"); do
   IFS=" ";
   for host in $hosts; do
      ssh -n -o BatchMode=yes "$host" './runScript.sh' 1>"$logdir/$host.log" 2>&1 &
   done
   wait
done

Assuming the 100s of hosts are listed one/line in the hosts.lst file and then from these a bunch are selected in one time (200), and on each of these 200 hosts are spawned your runScript.sh using ssh in batch mode and at the same time preserving the stdout+stderr spewing forth from each of these backgrounded job into a file with the name of host in the directory $logdir, which may be examined as and when required.

Finally we wait for one bunch to get over before we launch the next bunch, by means of the wait command at the end of the inner for loop.

  • Rakesh, i think you were trying to write 1000s ???? u wrote 100s. – api1411 Apr 12 '17 at 22:18
  • I just want to trigger the runscript only. so can " ssh -n -o Batch mode=yes "$host" " can be removed?? – api1411 Apr 12 '17 at 22:21
  • what do you mean trigger runscript.sh? If you want to remve ssh then why bother about that hosts list ? – user218374 Apr 12 '17 at 22:44
  • sorry "$host" is needed runscript.sh will take care of every thing like scp, rsync and all. I just need to trigger runscript.sh. – api1411 Apr 13 '17 at 0:51
  • Thanks Rakesh, It worked !! last question, Can i use exec in place of xargs?? What would be the impact.. – api1411 Apr 13 '17 at 4:31
2

The following modification to your script, which doesn't require any other utilities, will execute in parallel, but may run into limits depending on your hosts' limitations of open file handles:

for f in $(cat host.lst); do
       ./runScript.sh $f &
done

To capture the results to a log, just add > run.log after done to save to a new file, run.log.

The key change is the addition of &, which runs a process in the background rather than waiting for it to complete before executing the next command.

  • If the count is high in list suppose 1000, then it should pick the 200 one time..Because changing the list of host again will lead to manual task. – api1411 Apr 11 '17 at 23:35
  • you could use split to divvy up the list into batches, and do each batch in parallel. – DopeGhoti Apr 12 '17 at 16:25
  • in that case i have to maintain 10 for loop ( 10 batches). – api1411 Apr 12 '17 at 22:15

Not the answer you're looking for? Browse other questions tagged or ask your own question.