SSH hangs before login prompt

Question

I've got multiple reverse ssh tunnels set up from remote raspberry pis, running debian jessie. The RPIs use a 3G dongle to gain intrnet connectivity, which is why im using reverse ssh to remotely log in. Each RPI sets up a reverse ssh tunnel to a cloud server, which i use to log into each system.

The ssh tunnels are set up as follows:

ssh -N -o ExitOnForwardFailure=yes -R 23xx:localhost:22 username@178.x.x.xxx

where 23xx is the port used to forward connections from port 22, and 178.x.x.xxx is the ip address of the server.

My problem is that sometimes when i try to ssh into a system, it hangs forever, with no error as follows:

ssh pi_username@localhost -p 23xx

The terminal just does not output anything after that and hangs forever. When i try to debug using -vvv this is what i get:

ssh pi_username@localhost -p 23xx -vvv

OpenSSH_6.7p1 Debian-5+deb8u4, OpenSSL 1.0.1t  3 May 2016
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to localhost [::1] port 23xx.
debug1: Connection established.
debug1: key_load_public: No such file or directory
debug1: identity file /home/master/.ssh/id_rsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/master/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/master/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/master/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/master/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/master/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/master/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/master/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.7p1 Debian-5+deb8u4

It looks like the connection is established, but the login does not not get prompted. Any ideas? Any recommendation on how to further debug?

Answer 1

This is just a theory, but what I suspect is happening is that the TCP connections for your ssh sessions are dying, but the "cloud server" isn't detecting it. So when you go to do a connection to localhost -p 23xx, the ssh process is still there and listening, but when it tries to send data back to the Pi it hangs until the maximum number of TCP retransmits is encountered and finally decides the connection is dead and exits (you say it hangs forever, but I'm betting if you waited long enough you'd get a connection reset).
Now assuming you have the Pi configured to reconnect if the ssh tunnel dies, you might think this should take care of the issue. There are a couple potential issues with this idea. First is the Pi might also not be detecting the dead connection. So until it tries to send data across, and hits the TCP retransmits limit, it's not going to see the dead connection and do the reconnect. The second potential issue is that even if it does detect the dead connection and tries to reconnect, it's not going to be able to establish the listener on the cloud server because the previous ssh is still there and holding onto the port.

The solution here is to configure ssh so that it can detect the dead connections. There are a few ways to go about this, TCP KeepAlive and SSH KeepAlive. (ref: https://unix.stackexchange.com/a/34201/4358)

TCP KeepAlive (TCPKeepAlive setting in ssh config) uses TCP's native keep-alive functionality. Basically the kernel sends an empty TCP ACK every X seconds, and when it doesn't get an ACK back from the other end it (or it gets a reset) it closes the connection which notifies the app (SSH).

SSH KeepAlive (the ServerAlive* & ClientAlive* settings) is similar but operates at a higher layer. Here the SSH process sends actual data through the connection and looks for a response. This should detect a dead connection just as well as a normal TCP KeepAlive, however it's more likely to keep the connection alive as hops in the middle can recognize TCP KeepAlive packets and ignore them, and timeout the idle connection. But a SSH KeepAlive can't be recognized as it looks like real traffic to any hop in the middle.

TL;DR:

On the raspberry Pis, add the following settings to your ssh client config (~/.ssh/config or /etc/ssh/ssh_config):

ServerAliveInterval 15
ServerAliveCountMax 1

(documentation)

On the server, add the following settings to your ssh daemon config (/etc/ssh/sshd_config):

ClientAliveInterval 20
ClientAliveCountMax 1

(documentation)

Note that I made the Interval value slightly higher. The reason for this is just so that both sides don't send their KeepAlive messages at the exact same time and cross each other on the wire. There's no real harm in this, just a tiny inefficiency. It also doesn't matter which side is higher, just as long as they're different.