I have a remote controlling server which I use to automatically executing commands/scripts etc to all my nodes in series. To increase the performance and speed I wanted to run it in parallel. I am executing commands approximately in 180 nodes. In series it takes me around 2 hours to finish executing in all of them. So I am looking for speed at the moment. The pseodue code looks like the following, I used based64 to send the command to other nodes. I can execute any arbitrarily complex commands using it.

for host in ${sshconfig[@]}; do
 ssh "$host" "echo $COMMAND | base64 -d | bash" &

But I made a mistake forgetting that there were many nodes, so my bash script created many processes for each node, as a result, the control server's system had crashed and it has turned off. Now I cannot connect to it any more.

I would like to know, how can I allocate memory in a bash script? The ease of ssh has made bash really ideal for my situation but I was wondering how can I improve the performance by executing them in parallel and still not to worry about system crash etc.

  • How many hosts are you connecting to? What does your actual script look like? It's fairly easy to use xargs to do this in parallel in a more controlled manner. You may not allocate memory in a bash script (but the script you are showing does not look like bash at all).
    – Kusalananda
    Jul 30 '19 at 23:20
  • @Kusalananda Updated.
    – Rakib Fiha
    Jul 30 '19 at 23:46
  • 1
    are you running a cluster without a scheduler/workload & resource manager like slurm? it's packaged for ubuntu as slurm-wlm. If that's overkill for your needs, try pdsh (Parallel Distributed Shell - also packaged for ubuntu).
    – cas
    Jul 31 '19 at 0:56
  • I was thinking, to use services like chef, ansible, puppet but not willing to pay for their services. I will try out Pdsh! I was thinking how difficult it would be to make C or C++ based Programme that can allocate memories efficiently
    – Rakib Fiha
    Jul 31 '19 at 3:56

To limit the number of parallel SSH connections, you could use xargs like this:

printf '%s\n' "${sshconfig[@]}" |
xargs -P 4 -I {} ssh "{}" "echo $COMMAND | base64 -d | bash"

This would run your command in at most four parallel instances (given by -P 4). The -I {} option would make xargs replace the {} string in the command with a string read from its standard input. The standard input is connected to printf which simply prints one item from the sshconfig list at the time.

You would test run this with various -P settings until you found a number that seems to work well.

If your login shell on the remote machines is bash, you could also consider

code=$( base64 -d <<<"$COMMAND" )

printf '%s\n' "${sshconfig[@]}" |
xargs -P 4 -I {} ssh "{}" "$code"
  • I guess, in this case I do not require & anymore? I was also redirecting all my outputs 2>&1 locally, so I think with xargs I would be able to do that as well.
    – Rakib Fiha
    Aug 1 '19 at 3:57
  • @RakibFiha With xargs you would not use &. To redirect all output with 2>&1, place that redirection at the very end of the xargs command, or simply redirect the script itself from wherever you are running it.
    – Kusalananda
    Aug 1 '19 at 7:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.