I thought dsh(Distributed Shell) will be a good option but when I ran my shell script across my nodes, I didn't get my expected output,

dsh -aM -c bash /home/cloudera/bash_script.sh
server2@x.x.x.110: files in folder
server2@x.x.x.110: server2
server1@x.x.x.111: server1
server1@x.x.x.111: sleep time................
server3@x.x.x.112: server3
server3@x.x.x.112: sleep time................
server2@x.x.x.110: sleep time................



while true;

shopt -s nullglob
#shopt -s dotglob # To include hidden files
echo "files in folder" $files[@]

if [  ${#files[@]} -gt 0 ];
        for entry in "/home/cloudera/EcoAxis"/*
      cp $entry /home/cloudera/EcoBackup
      var=`basename $entry`
      var1=`echo ${var//[.csv]/}`
      echo $var1
      gawk -f abc.awk $entry
      rm -r -f $entry

 echo "server2" 
sleep 5s
 echo "sleep time................"
 sleep 10s

My script is working properly if I run it without dsh; why this abnormal behavior? By default, dsh can run up to 64 commands in parallel. Does dsh support all bash commands? What is the best option to go for parrallel processing of shell script accross nodes?

  • "server2" message is hardcoded, which does not match the output, I suppose scripts have the same name but are different on each servers ? – Emmanuel Aug 29 '14 at 14:59
  • have you considered porting it to Python Fabric? – thanasisk Aug 29 '14 at 15:05

Before you start porting try with GNU Parallel:

parallel -j0 --tag --line-buffer ssh {} bash /home/cloudera/bash_script.sh ::: server1 server2 server3

You need version 20130822 or later for --line-buffer.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.