How to prevent a program from dying when its ssh session dies?

Question

I have a really old console application that I want to make a bit more resilient. The program is used this way:

• the user uses a custom terminal emulator to connect to a remote machine through ssh
• the user starts a shell script
• the shell script might start a long-running progress database process.

Obviously, sometimes users simply lose the ssh connection to the machine, and in this case, the ssh session is terminated, the shell script running inside is terminated, and finally the progress database process is terminated as well. In like one case out of a thousand this causes corruption in the database, so I'd like to prevent it from happening.

What I tried so far:

• starting a screen or tmux session before starting the shell script - this doesn't work because the application needs the TERM variable to be set to at386 (and it bypasses termcap/terminfo completely... ugh...)
• nohup/disown the progress process - this doesn't work because the shell script and the progress process are continuously communication with each other in obscure ways it seems

Any other idea of how to make sure that the progress process doesn't terminate when the ssh session is killed?

Answer 1

Add term at386 to your .screenrc in order to override TERM. If this doesn't help, try dtach instead of Screen (which doesn't do any terminal emulation itself).

Answer 2

In many cases, the simplest solution is to use nohup and put the non-interactive process into the background using & and redirect standard output and standard error to files.

Answer 3

In other words, you're looking for something like screen, but that passes control sequences to the terminal rather than doing its own translation? Try dtach. It provides the same terminal detachment feature as screen, but without the multiple windows or the terminal emulation.

Answer 4

Perhaps try a remote X desktop via vnc or nx. You could tunnel vnc over ssh for security or use nx which runs via ssh by default (and is much faster). The app could then run in a Terminal on the X desktop and if the connection is lost, the desktop would keep going allowing reconnection at any time. My preference is No Machine NX which is free for up to two simultaneous users and installs very easily.

An alternative, if available, is to use a remote console via ALOM, ILOM, DRAC, IP-kvm or IP-serial console, etc.. That way the app would be running on the console and could not be interrupted by a connection failure.