3

I am trying to run code across a network of external nodes. I have access to the 'main' node through ssh, and can execute a parallel script that divides the jobs over the cluster of 5 available nodes. I have a bash script that contains the parallel command, among other necessary components.

The final command I am using looks similar to this.

parallel -S node0,node1,node2,node3,node4 --ssh-delay 0.25 --delay 0.5 'run {1} {2}' ::: foo ::: bar

However, the cluster that I am working on is known for hanging up relatively frequently, e.g., after ~5 minutes of idle time, I get a Broken pipe error, and the ssh- the connection is broken. It is why I execute the upper bash script (that contains the above parallel line) using nohup, which should keep the ssh-connection alive.

But because my actual code takes considerable computation time, I get errors related to broken ssh-connections:

ssh_askpass: exec(/usr/bin/ssh-askpass): No such file or directory^M
Permission denied, please try again.

even though connecting from (e.g. node0 to node1), the boss node to the other nodes does not require a password. This results in parallel complaining that there are no more job slots available and a warning that there are no logins possible:

parallel: Warning: There are no job slots available. Increase --jobs.
parallel: Warning: Using only -1 connections to avoid race conditions.
parallel: Warning: ssh to node0 only allows for 0 simultaneous logins.
parallel: Warning: You may raise this by changing /etc/ssh/sshd_config: MaxStartups and MaxSessions on node0.
parallel: Warning: You can also try --sshdelay 0.1

I believe that there is something finicky going on that prematurely closes the ssh-connection to the other nodes within the cluster, possibly the result of closing the connection to the boss node, node0.

I have tried to establish a connection using ssh-agent, ssh-copy-id, and sshpass as per the GNU parallel tutorial, as well as setting the MaxStartups and MaxSessions parameters in /etc/ssh/sshd_config, but to no avail. Even more, if I reduce the computation time of the code, then the parallel command is perfectly executed and works as expected.

Is there something I can do to ensure that the ssh-connection does not break when the program executed by parallel takes considerable time, or is there something else going on?

4
  • Am I correct that everything works if run takes less than 4 minutes to run and the problem only shows up with long running jobs?
    – Ole Tange
    Nov 10, 2019 at 1:11
  • How are you using ssh_askpass? Can you post the configuration that causes it to be called? GNU Parallel is not tested extensively with ssh_askpass so it there can easily be some issues with interaction between the two.
    – Ole Tange
    Nov 11, 2019 at 9:49
  • Yes indeed, the problem shows up only when running long jobs. I am not sure how ssh_askpass is used, since I do not call it myself directly. I assume it is called by parallel. Is there perhaps some configuration I can check? Nov 11, 2019 at 11:46
  • It is not called by GNU Parallel: it calls ssh. So something in your ssh-config causes this.
    – Ole Tange
    Nov 11, 2019 at 17:59

2 Answers 2

1

To keep the connection alive you can often use ServerAliveInterval. It can be set in .ssh/config.

3
  • I have indeed tried this and it seems that the server does not hangup anymore. However, I still get many warnings from GNU parallel that there are no job slots available: parallel: Warning: There are no job slots available. Increase --jobs. Nov 11, 2019 at 11:48
  • Can you increase to --sshdelay 5? I have the feeling something in your ssh connection causes it to overload.
    – Ole Tange
    Nov 11, 2019 at 18:02
  • Problems disappeared after putting the ServerAliveInterval on all nodes. Nov 12, 2019 at 12:41
1

I have found the origin of the error, namely that the ssh-key that was used to access the different nodes was incorrect. Thus, the problem was neither with GNU parallel nor with nohup.

Let me first illustrate the setup in a bit more detail, as to understand the problem. I first connect from my local computer to a 'boss'-node, boss which is connected to several slave nodes slaves. By enabling ssh-forwarding when using ssh to connect to the boss-node, further ssh-connection wrongly used the key from my local computer. Hence, when I broke the connection between the local and the boss, the boss had no access to the correct ssh-key in order to establish connections to the slaves nodes.

This problem was resolved by generating a new public ssh-key using ssh-keygen and distributing this key over the other nodes by adding it to ~/.ssh/authorized_keys. The result is that when the connection between the local and the boss is broken, the boss can still establish an ssh-connection to the slaves by using its own public key.

Since GNU parallel uses ssh to establish connections, once the keys were setup correctly, the permission (and other related) issues disappeared. Using nohup with setting the ServerAliveInterval 60 in /etc/ssh/ssh_config resolves all disconnection issues, and allows disconnection from the boss node while processes are kept running.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .