ssh command from another machine

Question

So I "am instructing" a command(which by the way is a script) within a script from machine A to machine B over ssh and sent it to background. The command goes something like this: ssh -T MACHINEB SCRIPT &

I do not have a timeout on the script that runs on machine A. Sometimes, I have noticed that the instruction of the command/script exists on the machine B (if I do a ps for it) but is kind of hanging. The ps shows correct PID and PPID but that is it, it does not complete.

I am trying to find what could cause the process to hang. I mean I am almost sure is code related, I just want to discard the following possibility and anything else you guys might add.

When I run an remote ssh script execution, is my process attached to the ssh session? So if I have a network or ssh problem from A to B, is my process affected?

Thanks

Answer 1

ssh -T MACHINEB SCRIPT & keeps the SSH connection running as long as SCRIPT is running. It needs to, because:

1. If SCRIPT produces any output, SSH must relay it to the host.
2. If SCRIPT reads input, SSH must relay it from the host. That can be a bit subtle.
3. When SCRIPT exits, SSH transmits its exit status to the client and the ssh process exits with this status.

If the connection goes down, that won't affect SCRIPT if it doesn't try to read or write to stdin, stdout or stderr. Since there's no terminal involved, SCRIPT won't be killed by a SIGHUP when the connection dies. However, if SCRIPT does try to, say, print an error message, that message goes through the connection.

You should do one of two things.

• Batch job method: make sure that SCRIPT doesn't take any input and is logging standard output and standard error to a file.

  ssh -T MACHINEB 'myprogram </dev/null >myprogram.log 2>&1'
  

• Interactive method: run SCRIPT inside a screen multiplexer: Screen or tmux.

  ssh -T MACHINEB screen -S somename -dm myprogram
  

  You can see how your program is doing by attaching to that screen session.

  ssh MACHINEB screen -S somename -dr
  

  The screen session exits when the program exits. To keep it around to see output and errors from the program, see Prevent GNU screen from terminating session once executed script ends

See also Execute remote commands, completely detaching from the ssh connection

Answer 2

Your question is not so clear and a bit confused, but I would try to provide some answers to clear things up for you. First, note that the ssh process runs on MACHINEA, while SCRIPT runs on MACHINEB. So the "&" sign puts the ssh command process on MACHINEA to the background - not the SCRIPT (shell) process you run on MACHINEB. So the (primary) script on MACHINEA continues to execute the following commands of the (primary) script, while MACHINEB runs within the "ssh" session - in the foreground as far as MACHINEB is concerned. So if you have a connectivity problem from A to B, your ssh command would not stop the primary script on MACHINEA - because the ssh process is running (possibly stuck) on the background.

Answer 3

When you run a script, it creates a subshell where the command executes. Any processes including background processes that are run, such as the ssh command are attached to this subshell. If the subshell is terminated, the background process is killed too. Instead, try running the ssh process as a nohup command within script 1. Answer inspired by: Start a background process from a script and manage it when the script ends

Commands sent through ssh are attached to the ssh process, and the ssh process must continue to run in order for the commands to complete. Exiting the ssh process will terminate any background processes on Machine B, since they're attached to the subshell created for script 2. Nohup comes in handy here as well.

But since your process is executing on Machine B but apparently hanging and not being terminated prematurely, it doesn't seem that a premature termination of the ssh process appears to be a problem.

Network issues are also unlikely since they would likely terminate the ssh process and terminate the processes running on Machine B. Additionally, if networking issues aren't so bad that terminate the ssh process, they shouldn't affect processes running on Machine B. There's likely something in script 2 on machine B that's causing the stall. One way to prove that this logic is wrong is to kill the ssh process on Machine A and see what happens on Machine B.

Finally, your best bet is to run your ssh command as a nohup process as suggested in the earlier link. Additionally, if your commands in script 2 are not sequentially dependent, I'd run each one using nohup as well. (https://askubuntu.com/questions/349262/run-a-nohup-command-over-ssh-then-disconnect). I'm assuming nothing in script 1 on machine A is dependent on script 2 executing based on the fact that you have the ssh command running as a background process.