What linux server settings allow my long running processes to keep running after the terminal is closed/terminated?

Question

In previous jobs and experiences, I used screen or nohup to run my long running jobs from my terminal screen. Most, if not all of my previous jobs' servers dropped you if it didn't detect any activity after X amount of time. Even if I had set ServerKeepAlive and other settings in my .ssh/config file, in some cases I still got booted, probably because the server ignored my settings. I used nohup once or twice but I mainly use screen to keep my process running.

Yesterday, I forgot to run my command with screen or nohup. I got booted off and I thought I was hosed. However, for some reason, the process was still running, even after several hours. I typed in ps -ef | grep myprocname and it was there until it was completed. I didn't run my process in the background (not sure if that makes a difference). I just type in ./script.sh and let it run.

What setting on the server allowed this fortuitous event to occur? I think the remote server is running Centos 6.4. Is this a Centos 6.4 thing, to not terminate processes when the parent process (in this case my ssh connection/shell) is killed? Do I need to go back to reading a Unix/Linux book since this is an easy question? Do I have my basic Unix facts wrong (about the parent process)? Did I get lucky?

Answer 1

This will not be answering your question in particular but be some general information on system behavior relevant to your problem and may help to understand how to better control what is happening.

In UNIX and UNIX-like systems processes started inside a shell will be child processes of that shell. If a shell closes (for example by disconnecting) or dies its child processes will receive a hangup signal (SIGHUP) and the child processes will be inherited by init, the first process, and thus will have the parent pid 1 like daemonized processes.

The child can implement a handler and it can be decided programatically what action to take. Usually the child will quit on hangup in shells.

To prevent that or take a different action you can use a signal handler(see trap in bash):

#!/bin/bash

trap '' HUP

# some long running action here

This will ignore the HUP signal. You can add some operations as well like this:

#!/bin/bash

on_hangup() {
    echo "Cought SIGHUP"
    # more commands/action
    # like redirection stdout to preserve output
    # or exit
}

trap 'on_hangup' HUP

# long running stuff here