0

So I "am instructing" a command(which by the way is a script) within a script from machine A to machine B over ssh and sent it to background. The command goes something like this: ssh -T MACHINEB SCRIPT &

I do not have a timeout on the script that runs on machine A. Sometimes, I have noticed that the instruction of the command/script exists on the machine B (if I do a ps for it) but is kind of hanging. The ps shows correct PID and PPID but that is it, it does not complete.

I am trying to find what could cause the process to hang. I mean I am almost sure is code related, I just want to discard the following possibility and anything else you guys might add.

When I run an remote ssh script execution, is my process attached to the ssh session? So if I have a network or ssh problem from A to B, is my process affected?

Thanks

1

ssh -T MACHINEB SCRIPT & keeps the SSH connection running as long as SCRIPT is running. It needs to, because:

  1. If SCRIPT produces any output, SSH must relay it to the host.
  2. If SCRIPT reads input, SSH must relay it from the host. That can be a bit subtle.
  3. When SCRIPT exits, SSH transmits its exit status to the client and the ssh process exits with this status.

If the connection goes down, that won't affect SCRIPT if it doesn't try to read or write to stdin, stdout or stderr. Since there's no terminal involved, SCRIPT won't be killed by a SIGHUP when the connection dies. However, if SCRIPT does try to, say, print an error message, that message goes through the connection.

You should do one of two things.

  • Batch job method: make sure that SCRIPT doesn't take any input and is logging standard output and standard error to a file.

    ssh -T MACHINEB 'myprogram </dev/null >myprogram.log 2>&1'
    
  • Interactive method: run SCRIPT inside a screen multiplexer: Screen or tmux.

    ssh -T MACHINEB screen -S somename -dm myprogram
    

    You can see how your program is doing by attaching to that screen session.

    ssh MACHINEB screen -S somename -dr
    

    The screen session exits when the program exits. To keep it around to see output and errors from the program, see Prevent GNU screen from terminating session once executed script ends

See also Execute remote commands, completely detaching from the ssh connection

1

Your question is not so clear and a bit confused, but I would try to provide some answers to clear things up for you. First, note that the ssh process runs on MACHINEA, while SCRIPT runs on MACHINEB. So the "&" sign puts the ssh command process on MACHINEA to the background - not the SCRIPT (shell) process you run on MACHINEB. So the (primary) script on MACHINEA continues to execute the following commands of the (primary) script, while MACHINEB runs within the "ssh" session - in the foreground as far as MACHINEB is concerned. So if you have a connectivity problem from A to B, your ssh command would not stop the primary script on MACHINEA - because the ssh process is running (possibly stuck) on the background.

  • Correct, I understand that is the ssh command that goes to background in the machine A. So my script on machine A continues his flow, and does not wait for an answer of the ssh command. The question exactly is: in the event of a network problem, and with the assumption that the ssh process that machine A launched to machine B exists on machine B(with the ps I can see it), does the network problem affects the already launched process? Or simply does not return successfully to the ssh background process executed on A? – Jonh Snow Aug 28 '17 at 15:29
  • So if I understand you correctly, you worry that after the SCRIPT on MACHINEB was started successfully over ssh from MACHINEA the connection between the machines would be interrupted and your SCRIPT on B would stop. If you wish to avoid this, you need to start the SCRIPT on MACHINEB in the background (the SCRIPT as background process on B - not ssh as background process on A). For example: ssh -T MACHINEB "SCRIPT &" (the & is inside the string of the command passed as part of the command to the ssh session on the target machine). – ElazarR Aug 28 '17 at 19:03
0

When you run a script, it creates a subshell where the command executes. Any processes including background processes that are run, such as the ssh command are attached to this subshell. If the subshell is terminated, the background process is killed too. Instead, try running the ssh process as a nohup command within script 1. Answer inspired by: Start a background process from a script and manage it when the script ends

Commands sent through ssh are attached to the ssh process, and the ssh process must continue to run in order for the commands to complete. Exiting the ssh process will terminate any background processes on Machine B, since they're attached to the subshell created for script 2. Nohup comes in handy here as well.

But since your process is executing on Machine B but apparently hanging and not being terminated prematurely, it doesn't seem that a premature termination of the ssh process appears to be a problem.

Network issues are also unlikely since they would likely terminate the ssh process and terminate the processes running on Machine B. Additionally, if networking issues aren't so bad that terminate the ssh process, they shouldn't affect processes running on Machine B. There's likely something in script 2 on machine B that's causing the stall. One way to prove that this logic is wrong is to kill the ssh process on Machine A and see what happens on Machine B.

Finally, your best bet is to run your ssh command as a nohup process as suggested in the earlier link. Additionally, if your commands in script 2 are not sequentially dependent, I'd run each one using nohup as well. (https://askubuntu.com/questions/349262/run-a-nohup-command-over-ssh-then-disconnect). I'm assuming nothing in script 1 on machine A is dependent on script 2 executing based on the fact that you have the ssh command running as a background process.

  • Ok got it. So I can assume it is not a network problem. Thanks – Jonh Snow Aug 28 '17 at 16:05
  • Edited my answer because it wasn't complete (lost some of my earlier answer in copying pasting after a log in). And as a brand new user, could totally use some karma if the answer was helpful. – user253385 Aug 28 '17 at 16:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.