Preventing terminal disconnection from killing a running job in zsh [duplicate]

Question

I went through the top answer to this question: Difference between nohup, disown and &, and I read specfically the following:

With disown the job is still connected to the terminal, so if the terminal is destroyed (which can happen if it was a pty, like those created by xterm or SSH, and the controlling program is terminated, by closing the xterm or terminating the SSH connection), the program will fail as soon as it tries to read from standard input or write to the standard output.

It looks like disown cannot prevent a job from being killed if the terminal closes. nohup can, but it can only be used when the job starts.

Assuming that I understood this correctly, what can I do to make sure that a job that I started without nohup does not get killed when I close my terminal, terminate my SSH connection?

---

Would issuing setopt NO_HUP from the command line do it? And if so, wouldn't that affect all running jobs that I started from the same terminal?

Answer 1

For a Standard Shell (bash) (POSIX.1)

Start it with &, make it read from something other then the default stdin (==/dev/tty == /dev/stdin == /dev/fd/0) + make it write to something other than the default stdout (==/dev/tty == /dev/stdin == /dev/fd/1) (same for stderr) and make sure the job isn't or doesn't get suspended(=stopped). If it must get stopped or must read from the terminal and it is to continue after the job's terminal hangs up, make sure the processes in the job have a handler (trap) for the SIGHUP signal. If it must write to the terminal and it is to survive a terminal hang up, make sure the processes that write to the terminal have a handler for SIGPIPE.

Explanation:

Background processes get sent SIGTTIN the moment they try to read from the terminal. The default disposition for SIGTTIN is to stop (=suspend) the process. A terminal hangup will cause the first generation of processes of your background jobs to get reparented to init, causing the process group of the first generation of your job's processes to become an orphaned process group. (Orphaned process groups are those where none of the members have a parent in a different process group but in the same session.) The system will send all stopped (suspended) orphaned process groups SIGHUP followed by SIGCONT (to make sure they get the SIGHUP) on a terminal hangup because by definition, none of the processes in that process group can be awoken by a parent from the same session. Consequently those processes must be awoken by the system, but at the same time in a way that signals that process that it got awoken due to the terminal hanging up rather than due to normal operation. SIGHUP is the mechanism that does that and the default disposition for SIGHUP is to abort.

Hanging up a terminal will also cause subsequent writes to that terminal to raise SIGPIPE, which is also deadly if unhandled.

TL;DR If your orphaned process groups aren't suspended (via SIGTTIN or ^Z or otherwise, they don't have to be afraid of the SIGHUP signal and if they output to a file rather than the terminal on both stdout and stderr and read from a file rather than the terminal, then they don't have to be afraid of SIGPIPE. If you're running on top of a terminal multiplexer rather than a raw terminal, youd don't have to be afraid of either SIGHUP or SIGPIPE.

---

For zsh

I played with zsh and it turns out that while bash does behave like I described above (the above described behavior should be conforming to POSIX.1), zsh sends SIGHUP to running background jobs even. setopt NO_HUP makes it (or the system; not sure here) only send SIGHUP in situations described above.

For testing this, you can try running something like:

( rm -f hup; trap 'echo HUP > hup' HUP; sleep 100)

in various ways (background, foreground, stopped) and disconnecting with it. Then you can check whether the hup file was created, which would mean the job did get a SIGHUP.

Answer 2

You may find useful (though it doesn't use disown, nohup or &) trying screen or tmux. Both tools allow you to run multiple terminals and detach from them without stopping what its happening in each one. I find tmux more convenient.

---

Using tmux

You can name sessions,

tmux new-session -s ${SESSION_NAME}

start one without attaching to it,

tmux new-session -d -s ${SESSION_NAME} '/home/user/script.sh'

list sessions,

tmux list-sessions

and even send commands to them:

tmux send-keys -t ${SESSION_NAME} "echo Hello world"
tmux send-keys -t ${SESSION_NAME} Enter

Of course you can attach to ("open") an ongoing session to see what is happening in there:

tmux attach-session -t ${SESSION_NAME}

And then leave it without closing it pressing Ctrl + B then D.

Answer 3

You could start your program as a batch job, e.g. using a here document :

batch << EOJ
  ./yourprogram some $ARGUMENTS
EOJ

If the program has been already started interactively, I don't see any general failproof solution; e.g. because that program might later start using system(3) (e.g. system("$EDITOR /tmp/somefile.txt");....) some X11 client application, or some interactive application using the tty(4)

If you can improve that program, you might consider having some program arguments enabling a call to daemon(3)

You could also use some terminal manager like screen(1) etc...