2

I've got multiple reverse ssh tunnels set up from remote raspberry pis, running debian jessie. The RPIs use a 3G dongle to gain intrnet connectivity, which is why im using reverse ssh to remotely log in. Each RPI sets up a reverse ssh tunnel to a cloud server, which i use to log into each system.

The ssh tunnels are set up as follows:

ssh -N -o ExitOnForwardFailure=yes -R 23xx:localhost:22 username@178.x.x.xxx

where 23xx is the port used to forward connections from port 22, and 178.x.x.xxx is the ip address of the server.

My problem is that sometimes when i try to ssh into a system, it hangs forever, with no error as follows:

ssh pi_username@localhost -p 23xx

The terminal just does not output anything after that and hangs forever. When i try to debug using -vvv this is what i get:

ssh pi_username@localhost -p 23xx -vvv

OpenSSH_6.7p1 Debian-5+deb8u4, OpenSSL 1.0.1t  3 May 2016
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to localhost [::1] port 23xx.
debug1: Connection established.
debug1: key_load_public: No such file or directory
debug1: identity file /home/master/.ssh/id_rsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/master/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/master/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/master/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/master/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/master/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/master/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/master/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.7p1 Debian-5+deb8u4

It looks like the connection is established, but the login does not not get prompted. Any ideas? Any recommendation on how to further debug?

Improve this question
4
  • Is the remote server under considerable load when this happens? – Kusalananda♦ Mar 5 '18 at 11:50
  • Hard question to answer. It is a cloudways managed server, to which i do not even have root access. Do you think that could be causing the problem? – LecauseAndThePi Mar 5 '18 at 11:55
  • Well, it's a possible explanation. The server is so busy it can't respond properly to the SSH connection to do authentication and open a shell. – Kusalananda♦ Mar 5 '18 at 11:56
  • Any more answers for this? Just began on an up-to-date Debian server, logs show nothing. – Jortstek Sep 1 '19 at 17:18
3

This is just a theory, but what I suspect is happening is that the TCP connections for your ssh sessions are dying, but the "cloud server" isn't detecting it. So when you go to do a connection to localhost -p 23xx, the ssh process is still there and listening, but when it tries to send data back to the Pi it hangs until the maximum number of TCP retransmits is encountered and finally decides the connection is dead and exits (you say it hangs forever, but I'm betting if you waited long enough you'd get a connection reset).
Now assuming you have the Pi configured to reconnect if the ssh tunnel dies, you might think this should take care of the issue. There are a couple potential issues with this idea. First is the Pi might also not be detecting the dead connection. So until it tries to send data across, and hits the TCP retransmits limit, it's not going to see the dead connection and do the reconnect. The second potential issue is that even if it does detect the dead connection and tries to reconnect, it's not going to be able to establish the listener on the cloud server because the previous ssh is still there and holding onto the port.

The solution here is to configure ssh so that it can detect the dead connections. There are a few ways to go about this, TCP KeepAlive and SSH KeepAlive. (ref: https://unix.stackexchange.com/a/34201/4358)

TCP KeepAlive (TCPKeepAlive setting in ssh config) uses TCP's native keep-alive functionality. Basically the kernel sends an empty TCP ACK every X seconds, and when it doesn't get an ACK back from the other end it (or it gets a reset) it closes the connection which notifies the app (SSH).

SSH KeepAlive (the ServerAlive* & ClientAlive* settings) is similar but operates at a higher layer. Here the SSH process sends actual data through the connection and looks for a response. This should detect a dead connection just as well as a normal TCP KeepAlive, however it's more likely to keep the connection alive as hops in the middle can recognize TCP KeepAlive packets and ignore them, and timeout the idle connection. But a SSH KeepAlive can't be recognized as it looks like real traffic to any hop in the middle.

TL;DR:

On the raspberry Pis, add the following settings to your ssh client config (~/.ssh/config or /etc/ssh/ssh_config):

ServerAliveInterval 15
ServerAliveCountMax 1

(documentation)

On the server, add the following settings to your ssh daemon config (/etc/ssh/sshd_config):

ClientAliveInterval 20
ClientAliveCountMax 1

(documentation)

Note that I made the Interval value slightly higher. The reason for this is just so that both sides don't send their KeepAlive messages at the exact same time and cross each other on the wire. There's no real harm in this, just a tiny inefficiency. It also doesn't matter which side is higher, just as long as they're different.

Improve this answer
3
  • Thank you for the very detailed reply. It all makes sense. I will make the chanegs and test on this. The only problem may be trying to make changes to the server side of things, since cloudways does not allow root access. Thanks again. – LecauseAndThePi Mar 7 '18 at 10:05
  • Is there any point making these changes to rpi side only? – LecauseAndThePi Mar 7 '18 at 10:27
  • You can try it. Maybe the server has tcp keep alive enabled by default, which would be good enough. But if it doesn't, then I expect you'll still have issues. – phemmer Mar 7 '18 at 13:02
0

I got the same issue.

$ ifconfig wlan0
wlan0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.1.9  netmask 255.255.255.0  broadcast 192.168.1.255
        inet6 fe80::d949:d257:c622:3388  prefixlen 64  scopeid 0x20<link>
        ether 18:1d:ea:00:4e:cf  txqueuelen 1000  (Ethernet)
        RX packets 333970  bytes 471184965 (449.3 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 76462  bytes 9787485 (9.3 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Change mtu value to 1200. (MTU)

$ ifconfig wlan0
wlan0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1200
        inet 192.168.1.9  netmask 255.255.255.0  broadcast 192.168.1.255
        ether 18:1d:ea:00:4e:cf  txqueuelen 1000  (Ethernet)
        RX packets 334117  bytes 471271158 (449.4 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 76604  bytes 9808842 (9.3 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

SSH connected as expected after this. Hope this answer shortens your search for workarounds. :)

Improve this answer
2
  • If lowering the MTU addresses the issue, the proper fix is to stop blocking ICMP packets. – phemmer Feb 1 at 22:15
  • eventually this is the bug in Arch Linux or SSH itself. I didn't encounter this in Fedora 33. MTU 1500 is generally set due to the recommendation in IPv6, so setting it around 1400 is best for both IPv4 and IPv6 – Rahul Bali Feb 2 at 6:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.