18

I often have to connect to a server over ssh in an unreliable wifi environment. On the server, I run screen, so if I get disconnected, I can reconnect and resume the screen session, and pick up where I left off, but the loss of connection is still a major time sink: if the connection drops out while I'm on the server, the terminal window tends to freeze. I have to kill that tab, open a new one, ssh to the server again and resume the screen session. I've tried this with running screen on the server and screen locally. Either way it tends to freeze when the connection drops out.

Is there any way I can have something similar to screen, or maybe screen itself, that will automatically try to reconnect and keep the session running, so I don't have to keep manually reconnecting? Often when I lose the connection I think it's only for a very brief period - less than a second maybe.

I'm using Ubuntu 14.04 LTS, MATE edition. thanks

  • 4
    Re "the shell window tends to freeze": That's because your local ssh doesn't know the connection is dead. Hit <Enter> and type ~. to tell your side to drop the connection, and you can simply repeat the last ssh command to reconnect (e.g. with up-arrow or !!). – alexis Jun 4 '18 at 20:41
  • @alexis that sounds like a quicker way to reconnect, thanks! I'd love it to happen automatically though... – Max Williams Jun 5 '18 at 8:36
23

You could look at using mosh: https://mosh.org/

You could set up a 'jump' server with a reliable internet connection which you use mosh to connect to, then have ssh sessions to each server you manage. The reason I suggest using a jump server is that you may not wish to install mosh on the servers you are managing.

Another advantage of mosh is that it is based on UDP rather than TCP, and your session can survive a change of IP address, for example going from WiFi to a mobile internet connection.

Just to make it clear, mosh is not a replacement for screen, but rather ssh. It's still a good idea to use screen with it, since mosh itself doesn't provide a way to reconnect to your session if the client dies for some reason.

  • Thanks, it is just the one server (most of the time) and we own it so I should be able to install Mosh. I'll check it out. – Max Williams Jun 4 '18 at 7:59
  • Actually it turns out that because our server's quite old (or running an old Ubuntu i should say) it's too difficult to install. :( – Max Williams Jun 4 '18 at 8:51
  • @MaxWilliams how old is it? Even LTS 12.4 has gone out of support. And why don't just try compiling it yourself – phuclv Jun 4 '18 at 9:33
  • As I read the mosh docs, you need mosh-server on each host you intend to remotely manage. Still, definitely interesting. – Wildcard Jun 4 '18 at 9:43
  • 1
    Connecting to a tmux terminal over mosh is the most stable solution for me. – Nemo Jun 4 '18 at 11:47
3

I have been using tmux for a few years now and in my experience, it reconnects automatically. At least when the connection only fails for a relatively brief time. Note that I actually use byobu with tmux as the backend. I don't know if this robustness is a feature of tmux or byobu or even of the combination of the two, but I suggest you give both a try.

I connect from my local Arch install to various remote Ubuntu servers through a VPN. I tested it just now by unplugging my network cable while I was connected to the remote. The session hung, but as soon as my cable was plugged in again, it resumed seamlessly.

However, when I tested by restarting my router, the connection did not return. I assume it has something to do with how long the network was down, but it does seem to reconnect if it's only down a few seconds.

In case it's relevant, I do all this using terminator as my terminal emulator.

All three are available in the Ubuntu repositories:

sudo apt-get install tmux terminator byobu

However, I am not at all sure that either tmux or byobu are better at handling ssh disconnections. I only know that in my experience, they often come back from short connection losses. That may be down to other aspects of my configuration though.

  • 1
    When you restarted your router you may have been given a different public IP address, which will have broken the tcp connection. From my experience ssh can be very resilient to intermittent network drop outs, I don't think this has anything to do with the fact you're using tmux inside the ssh window. – rusty shackleford Jun 4 '18 at 10:31
  • 3
    I was going to say the same: even with plain SSH, you can deal with a short disconnection, as long as the TCP connection doesn't die. Which it might, if your interface gets shutdown, or some overzealous router kills it (NAT routers might forget the NAT state on reboot, and break existing connections), or ClientAlive/ServerAlive triggers, or... I've no idea what byobu does, though. – ilkkachu Jun 4 '18 at 10:34
  • Yes, but the OP seems to be experiencing freezing on any connection failure, while I don't. But yes, you're right, I also see this with simple ssh and no tmux. Nevertheless, maybe screen can't deal with it? – terdon Jun 4 '18 at 10:56
  • 2
    @MaxWilliams tmux is basically a more modern alternative to screen, yes. When I first started to work as I do now and needed this sort of thing, my cursory reading suggested that tmux is the better choice these days. I am also not 100% sure it has better management of lost connections, all I know is that it does recover from brief outages in my experience. Whether that's down to tmux or something else, I don't know. But it seems worth trying :). Byobu is basically a frontend to screen/tmux, not a GUI terminal emulator. It is exceedingly useful though: byobu.org – terdon Jun 4 '18 at 12:56
  • 2
    tmux does not do anything about connection interruptions. It works with the terminal device provided by ssh. It all stands and falls with the ssh connection. – Jonas Schäfer Jun 4 '18 at 16:03
2

Use ssh's ServerAlive options to detect when the connection has failed.

ServerAliveCountMax
Sets the number of server alive messages (see below) which may be sent without ssh(1) receiving any messages back from the server. If this threshold is reached while server alive messages are being sent, ssh will disconnect from the server, terminating the session. It is important to note that the use of server alive messages is very different from TCPKeepAlive (below). The server alive messages are sent through the encrypted channel and therefore will not be spoofable. The TCP keepalive option enabled by TCPKeepAlive is spoofable. The server alive mechanism is valuable when the client or server depend on knowing when a connection has become inactive.

The default value is 3. If, for example, ServerAliveInterval (see below) is set to 15 and ServerAliveCountMax is left at the default, if the server becomes unresponsive, ssh will disconnect after approximately 45 seconds.

ServerAliveInterval
Sets a timeout interval in seconds after which if no data has been received from the server, ssh(1) will send a message through the encrypted channel to request a response from the server. The default is 0, indicating that these messages will not be sent to the server.

So if you set ServerAliveInterval to 5, ssh will automatically disconnect if the network flakes out for 15 seconds.

  • To break an SSH session by force, I press ~. (or first Enter, then ~.) consisting of: the escape character ~ and the command to break the session . – imz -- Ivan Zakharyaschev Jun 4 '18 at 22:15
  • @imz--IvanZakharyaschev That assumes you can tell that the connection is hung. Using SSH's keepalive will detect the failure automatically. – Barmar Jun 4 '18 at 23:42
  • That sounds really useful, thanks, I'll definitely try that next time i'm in the "flaky zone". – Max Williams Jun 5 '18 at 7:24
  • @Barmar Yes, true. I've also thought about the problem of determining whether the connection is really hung, or me pressing something can accidentally send these keys to the remote side... And I don't know a good solution. – imz -- Ivan Zakharyaschev Jun 6 '18 at 10:20
2

In similar conditions, I tend to use eshell with TRAMP (over ssh) inside Emacs. TRAMP takes care of reconnecting when necessary without causing much trouble for me giving the wanted commands for the remote shell.

However, eshell is not good as a terminal, i.e., for running commands that do something special with the terminal, or that run for a significant period of time continuously (incrementally) printing something out.

Basically, it's quite simple to start using it in Emacs with TRAMP:

M-x eshell
cd /user@host:
1

Disclaimer

If your SSH connection is not surviving brief network outages, then there's something else going on that's not letting ssh and TCP do their normal thing.

See below for details. Anyway:

The Quickest-and-Dirtiest No-Dependencies Solution

Create a shell script like this:

#!/bin/sh -

# Tune these numbers depending on how aggressively
# you want your SSH session to get reconnected.
timeout_options='-o ServerAliveInterval=4 -o ServerAliveCountMax=2'

# 255 is the status OpenSSH uses to signal SSH errors, which
# means we want to connect. All other exit statuses suggest
# an intentional exit.
status=255

# Keep opening the SSH connection and immediately dropping into
# `screen` until an intentional exit happens.
while [ "$status" = 255 ]
do
    ssh $timeout_options -t "$@" screen -dR
    status=$?
    # You can add a `sleep` command here or a counter or whatever
    # you might need as far as rate/retry limiting.
done
exit "$status"

This will just run a stupid-simple loop that keeps trying to connect with ssh and attach to screen. Pass the host or whatever else you'd normally pass to your ssh invocation as command-line arguments.

The reconnect is just based on whether SSH reports an error with the connection, which means it has no intelligence for detecting non-SSH errors like "you literally don't have WiFI turned on" or whatever, but that probably doesn't matter for you.

I'm assuming you have ssh-agent or a no-passphrase SSH key that will allow reconnects to just work without additional input from you.

There's going to be a tiny race condition where if you hit ^C during just the right human-imperceptible fraction of a second during a reconnect you could end up killing the script instead of passing the ^C through to the client terminal, so if you suspect a connection hang don't mash ^C too zealously.

Simplest Additional Software Solution

You can try the program autossh, which should be available in your Ubuntu package repository.

If you need to build from source or audit it, it's a single C program that compiles without any additional libraries as dependencies, seems to have more intelligence about checking connection liveliness than my hack above, and it also ships with a convenient rscreen script command which auto-attaches to screen.

Details

How ssh normally recovers

Just to verify, because I don't like saying things without checking myself, I ran a little test before answering:

I got on my WiFi with a Linux device, made an SSH connection to another device on my LAN, verified I had a working ssh connection to the other end (could run commands, etc), then on the client disconnected the WiFi (causing the interface to be de-configured: no more IP addresses), typed a bunch more characters into the ssh session (no response, of course), and then reconnected to my WiFi - the reconnection actually failed at least once due to bad signal and other factors, then finally reconnected: I waited about five seconds for the ssh session to recover, nothing happened so I hit one more key, and the ssh session immediately came alive again, with all the keys that I had typed during the disconnect appearing on the command line.

See, ssh just writes/reads into the TCP network socket until the OS tells it something went wrong, and TCP is actually very tolerant of prolonged connection drops.

Left to its own devices with default kernel settings the TCP stack in Linux will happily tolerate the connection going completely silent for many minutes before declaring the connection dead and reporting an error to ssh - by the time it finally gives up we're talking in the ballpark of ~30 minutes, or at least certainly long enough to outlast connection hiccups lasting a second or a minute.

Underneath the covers, the Linux TCP stack gradually retries messages with longer and longer delays, though, which means that by the time your connection does come back, you might be looking at additional lag before your ssh session seems to come "alive" again.

Why this sometimes breaks

Often something is actively causing the connection to close after some significantly shorter period of inactivity than the amount that the TCP stack will tolerate, and then failing to report that connection state to your ssh client.

Likely candidates include:

  1. Firewalls or NAT'ing routers, which have to use memory to remember each live TCP connection - as an optimization and some mitigation against DOS attacks, they will sometimes just forget your connection, and then silently ignore consequent packets for it, because packets in the middle of a connect when you don't remember the connection existing look invalid.

  2. Better-behaved firewalls/routers will inject a TCP RST packet, which typically manifests as a connection reset by peer error message, but the reset packet is a fire-and-forget, so if the connection to your client is still having issues at that moment and drops the reset packet too, your client will think the connection is still alive.

  3. The server itself might have a firewall policy to silently drop unexpected packets, which would break the client's connection resumption attempts whenever the server thinks the connection closed but the client doesn't: your client keeps trying to continue the connection, but the server is just ignoring it because there's no live connection to which these packets belong in the server's firewall state.

    Since you're running Linux, carefully check your server's iptables/ip6tables (or nft if you're using the new stuff) for exactly what you're allowing vs. dropping. It's very common to allow new/established/related packets on the TCP SSH port, but not "invalid" ones - if you're dropping silently everything that isn't allowed, this common setup could cause these kinds of freezes after brief connection issues.

  4. Your SSH server itself might be configured to close the connection after a period of inactivity, using one of OpenSSH options for TCP or SSH client keepalive packets. By itself this won't cause indefinite hangs, but it can put you in one of the states described above.

  5. It's possible you're just not giving it enough time to "unhang" on its own after you get into the state where your ssh session hangs up.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.