28

Does a program that is run from an ssh session depend on the connection to the client? For example when the connection is really slow. So does it actively wait until things are printed on the screen?

And if it does depend on the connection, does it also happen with screen or byobu for example? Since with these the programs are kept running even after disconnecting from the host.


Note: I only found these related questions:

  • 1
    Note that if your connection is lost the program will continue running until the session timeouts (unless its stuck in a print operation), but if your session ends the program will be terminated gracefully. – Sebb May 13 '16 at 17:09
25

The output of programs is buffered, so if the connection is slow the program will be halted if the buffer fills up.

If you use screen, it has a buffer as well that it uses to try and display to a connected session. But a program connected in the screen session will not be stopped if screen cannot update the remote terminal fast enough. Just like when a connection is lost, the program continues filling screens buffer until it overflows (pushing out the oldest information). What you see coming in (and can scroll back to) is depending on what is (still) in that buffer. screen effectively discouples your program from your terminal (and your slow SSH connection).

9

An SSH connection can die prematurely if the underlying TCP connection receives a packet with the RST flag. That could happen if one side sends a packet (which might be a periodic SSH keepalive probe) but doesn't receive a TCP acknowledgement in a reasonable amount of time, or if a router decides that the connection has been idle too long, or if an ISP is just being evil.

In the Unix terminal model, when the terminal connection is dropped, the terminal driver sends a HUP signal to the shell, whose termination also causes a SIGHUP to be sent to processes running in the shell.

From the Unix Programmer FAQ, item 1.15:

SIGHUP is a signal that means, by convention, "the terminal line got hung up". It has nothing to do with parent processes, and is usually generated by the tty driver (and delivered to the foreground process group).

However, as part of the session management system, there are exactly two cases where SIGHUP is sent on the death of a process:

  • When the process that dies is the session leader of a session that is attached to a terminal device, SIGHUP is sent to all processes in the foreground process group of that terminal device.

  • When the death of a process causes a process group to become orphaned, and one or more processes in the orphaned group are stopped, then SIGHUP and SIGCONT are sent to all members of the orphaned group. (An orphaned process group is one where no process in the group has a parent which is part of the same session, but not the same process group.)

The default signal handler for SIGHUP is to terminate the process:

Signal     Value     Action   Comment
----------------------------------------------------------------------
SIGHUP        1       Term    Hangup detected on controlling terminal
                              or death of controlling process

It is possible to avoid process termination, though.

  • You can insert a signal handler that ignores SIGHUP. To do that as a user, wrap the command in nohup. For example:

    nohup make all &
    
  • You can tell the shell to dissociate a child process from it. For example, Bash has a disown built-in command:

    make all
    

    CtrlZ

    bg
    disown %1
    

    Then, the SIGHUP will not be propagated to the child (which is no longer a child).

  • Some programs, notably daemons, will use the mechanisms above automatically: a program can install an alternate SIGHUP handler (using sigaction(2)), or it could choose to join a new session (setsid(2)).
  • You can run screen or tmux, which allocates a pseudo-TTY to run a session with a shell that does not receive the SIGHUP when the SSH connection dies. The SIGHUP is not relayed from the SSH session to the screen/tmux session.

Incidentally, an alternate way to deal with unreliable SSH connections is to use the Mosh protocol instead. Mosh runs over UDP, so there is no TCP connection that risks getting reset.

1

Yes, a program running over SSH will depend on its output going somewhere. If the connection is slow, the output must be buffered somewhere, and buffers cannot be infinite, so the program must block if they are filled.

Note that the output might not necessarily go to a terminal: consider running something like

ssh user@somewhere "cat file.txt" > file.txt

This will in effect copy the file. For this to work, the output rate of cat must match that of the connection: it should be obvious that losing parts of the output from the middle would be unacceptable.

Screen will change the situation in that it acts like a terminal and will save what should be shown "on the terminal window" (plus scrollback). It doesn't need to remember everything your program outputs, only the parts that will fit the "window" and scrollback. By default, screen will wait for a slow connection (blocking the program), but it can be configured to detect a stuck connection by setting "nonblock on".

From the man page:

nonblock [on|off|numsecs]

Tell screen how to deal with user interfaces (displays) that cease to accept output. This can happen if a user presses ^S or a TCP/modem connection gets cut but no hangup is received. If nonblock is off (this is the default) screen waits until the display restarts to accept the output. If nonblock is on, screen waits until the timeout is reached (on is treated as 1s). If the display still doesn't receive characters, screen will consider it "blocked" and stop sending characters to it. If at some time it restarts to accept characters, screen will unblock the display and redisplay the updated window contents.

A disconnection is different than a slow connection. Plain SSH cannot recover from it automatically, so your program will receive a SIGHUP. On the other hand, screen will detect a disconnection, detach and fall back to local buffering until the screen is reattached. This will not block the running program.

(Setting nonblock 1 in your .screenrc is important if you run something like irssi that will continously produce output but still must talk to the network at the same time. Blocking would lead to getting disconnected from IRC, which is extremely annoying...)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.