persistent ssh sessions without screen or tmux

Question

My work seems to have deliberately crippled screen and tmux such that they destroy all sessions on logout (they didn't answer when I opened a ticket about it). i.e., when I [detach and] log out, either with ctrl-D or just close the terminal emulator on my local machine, and log back in, the tmux and screen no longer have any detached sessions.

1) Is there a way around this?

2) If the system is killing all background processes for logged out users I can't expect to recover an accidentally disconnected session, but is there a tool for writing details of the session to the disk? Such as there is in emacs, for example?

I don't know what details to provide yet, so please ask.

Answer 1

Aside from running emacs -nw via ssh (and having it save its session), your options are slim to none, because very few tools do this (outside of the GUI desktops, whose attempts to do this are frequently bizarre).

Since your "work" wants to kill idle sessions, they've probably also limited or eliminated cron.

You could try nohup, but usually that's checked-out and killed as well. Any competent systems administrator has already forestalled the workarounds which are available to you.

Answer 2

There's a solution just for that: Mosh!

Mosh uses UDP instead of TCP and allows the client to change IP addresses and continue the session from a different location or after an event that would have broken a TCP connection. It also tracks the contents of the screen (like screen and tmux does) on the server so the server can send the necessary screen updates to bring the client up to date if it has missed any updates while it was disconnected.

Mosh piggybacks on SSH to initially log in and set up the session, but after that, it uses its own protocol over UDP.

Answer 3

Both @Thomas Dickey and @Celada have good suggestions, and after messing around with this for a few days I have an ok solution that may be of interest.

First: Certainly you should use Mosh if that is available server-side. It will allow you to work with a faulty / slow connection.

Second, use an editor that will save your work to a desktop - emacs has this, for example. According to the documentation and the variable desktop-auto-save-timeout, it will autosave every 30 seconds by default in Emacs 24.4 when using (desktop-save-mode 1) in your init file. Note that emacs may lock a desktop file so emacs restarted after crash will not successfully do (desktop-read). Some code from here helped me out (but I don't know how it works). Add it to your init file:

;;; desktop-override-stale-locks.el begins here
(defun emacs-process-p (pid)
    "If pid is the process ID of an emacs process, return t, else nil.
Also returns nil if pid is nil."
    (when pid
      (let ((attributes (process-attributes pid)) (cmd))
        (dolist (attr attributes)
          (if (string= "comm" (car attr))
              (setq cmd (cdr attr))))
        (if (and cmd (or (string= "emacs" cmd) (string= "emacs.exe" cmd))) t))))

(defadvice desktop-owner (after pry-from-cold-dead-hands activate)
  "Don't allow dead emacsen to own the desktop file."
  (when (not (emacs-process-p ad-return-value))
    (setq ad-return-value nil)))
;;; desktop-override-stale-locks.el ends here

I use tmux with tmuxinator client-side which opens a few windows/panes that ssh to my working directory server-side. It's not perfect, but it's still fairly efficient to setup the work environment with instances of htop, and various open directories using my faviourite shell. A few useful things in my tmux config file:

# make sure your terminal emulator and tmux are on the same colour scheme, or your nice themes will get screwed up.
set -g default-terminal "xterm-256color"
#if you use vim in tmux you need this for quick ESC key sequences, else it gobbles up ESC key presses.
set -s escape-time 0
# Enable mouse mode (tmux 2.1 and above)
set -g mouse on
# don't rename windows automatically.  useful for window "categories"
set-option -g allow-rename off
# switch panes using Alt-arrow without prefix
bind -n M-Left select-pane -L
bind -n M-Right select-pane -R
bind -n M-Up select-pane -U
bind -n M-Down select-pane -D

And a snippet of my tmuxinator yml config file

windows:
  - ouput: # this is the name of a window
      layout: tiled
      panes: # these are some panes in said window
        - cd /some/directory               
        - cd /another/directory            
        - cd ~/
        - cd ~/
  - remoutput:
      layout: tiled
      panes:
        # cd to working dir and start up a zsh session
        - ssh me@host -t 'cd ~/working/dir; zsh -i'
        # repeat more panes and windows server-side

Upon disconnect the tmux windows with ssh will exit to a local shell where the previous command was the tmux comment, so at least it's easy to up+RET back to the server. You'll want to avoid "shared incremental history" options in your environment to make sure this works.

Finally, if your system is determined to kill idle sessions, just have your client send an SSH poke every minute or so so they don't realise you are idle. Do this in the .ssh/config with

Host *
    ServerAliveInterval 60

I deliberately left emacs out of the tmuxinator configuration for two reasons. 1 tmux can be slow with text editors, struggling to refresh so much text and 2 I wanted to run emacs in server-mode and it might already be running when I start up in tmux. I would generally run separate windows, client and server-side, dedicated to emacs (with the same configuration, of course) running in server-mode. In addition, I set EDITOR="emacsclient -t", so that quick editing tasks will instantly connect to the emacs server wherever they are called from. All of these tasks, of course, become part of the emacs desktop! (if anyone wants to elaborate on how to do this with the other editor, I'm interested.)