0

I thought dsh(Distributed Shell) will be a good option but when I ran my shell script across my nodes, I didn't get my expected output,

dsh -aM -c bash /home/cloudera/bash_script.sh
[email protected]: files in folder
[email protected]: server2
[email protected]: server1
[email protected]: sleep time................
[email protected]: server3
[email protected]: sleep time................
[email protected]: sleep time................

bash_script.sh

#!/bin/bash

while true;
do

shopt -s nullglob
#shopt -s dotglob # To include hidden files
files=(/home/cloudera/MyFolder/*)
echo "files in folder" $files[@]

if [  ${#files[@]} -gt 0 ];
then
        for entry in "/home/cloudera/abc"/*
    do
      cp $entry /home/cloudera/Backup
      var=`basename $entry`
      var1=`echo ${var//[.csv]/}`
      echo $var1
      gawk -f abc.awk $entry
      rm -r -f $entry

    done
fi
 echo "server2" 
sleep 5s
 echo "sleep time................"
 sleep 10s
done

My script is working properly if I run it without dsh; why this abnormal behavior? By default, dsh can run up to 64 commands in parallel. Does dsh support all bash commands? What is the best option to go for parrallel processing of shell script accross nodes?

2
  • "server2" message is hardcoded, which does not match the output, I suppose scripts have the same name but are different on each servers ?
    – Emmanuel
    Aug 29, 2014 at 14:59
  • have you considered porting it to Python Fabric?
    – thanasisk
    Aug 29, 2014 at 15:05

1 Answer 1

2

Before you start porting try with GNU Parallel:

parallel -j0 --tag --line-buffer ssh {} bash /home/cloudera/bash_script.sh ::: server1 server2 server3

You need version 20130822 or later for --line-buffer.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .