2

I have two systems: An Ubuntu 20.04 server on the Internet and a Raspberry Pi 2B (Ubuntu 18.04) in my local LAN (behind a NAT router). I am opening a reverse tunnel from the Raspberry Pi to the server like so:

ssh -f -N -R 13333:127.0.0.1:22 server

The config file looks like:

Host server
  Hostname <public IP>
  User serveruser
  IdentityFile /home/piuser/.ssh/serverkey

Then on the server I am running the following command:

ssh -p13333 -i ~/.ssh/pikey piuser@127.0.0.1

For the pikey, a command is executed on the Raspberry (set in the authorized_keys file).

I tried this setup a few times yesterday, but today (~24h later) I couldn't use the tunnel anymore. ss showed me the open ports, and I was able to execute the command on the server, but there was no response- the command just "hung".

Then, I killed the tunnel and re-established it, without any problem. To troubleshoot the problem more, I added -vvv to the command sent from the server:

ssh -vvv -p13333 -i ~/.ssh/pikey piuser@127.0.0.1

This showed me the stuck connection at the following point:

OpenSSH_8.2p1 Ubuntu-4ubuntu0.2, OpenSSL 1.1.1f  31 Mar 2020
debug1: Reading configuration data /home/serveruser/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug2: resolve_canonicalize: hostname 127.0.0.1 is address
debug2: ssh_connect_direct
debug1: Connecting to 127.0.0.1 [127.0.0.1] port 13333.
debug1: Connection established.
debug1: identity file /home/serveruser/.ssh/pikey type 0
debug1: identity file /home/serveruser/.ssh/pikey-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.2

This time, on the site of the Raspberry Pi I saw:

$ Warning: remote port forwarding failed for listen port

So my last try was rebooting the Raspberry, and, voila after establishing the tunnel again, it worked as it's supposed to work.

Finally, my question: How can I determine what went wrong? I rely on that connection when abroad, so I really need to figure out what went wrong and how I can fix that...
Btw, I Googled a bit, but none of the suggestions / solutions work for my case, since I was able to establish the connection in the first place, but some hickup brought it into a strange state.

If you need anymore information, please let me know :)

2

It looks as though you might need to enable keepalive functionality. Something like the following in your .ssh/config might work:

Host *
    ServerAliveInterval 300
    ServerAliveCountMax 2
1
  • Thanks! I will try that and wait for 24h+ and then report back :) – JJ Abrams Apr 9 at 14:34
2

You will get Warning: remote port forwarding failed for listen port when there is already something listening on the specified port. You can use sudo netstat -lnp | grep 13333 if this happens again to check if something is still listening on that port. The p flag on netstat should show you the pid of whatever is listening on that port.

Chances are that your ssh tunnel will end up timing out if there is nothing using it for an extended period of time. You could try using autossh as a utility to monitor the tunnel and re-establish it when it fails. This superuser post shows how to do so.

5
  • 1
    Thanks! I will first try what Rory suggested in the post above and if that fails, autossh. That way, I don't need to install a new package right away :) – JJ Abrams Apr 9 at 14:35
  • 1
    Sounds good! If you do run into the same issue with the remote port forwarding failing, try out the netstat command. I think you should be able to recover from that without rebooting the raspberry pi. – Natolio Apr 9 at 14:41
  • I will, thanks man :) – JJ Abrams Apr 9 at 15:18
  • 1
    Good idea to 'cradle' (auto restart) the ssh command like you're suggesting with autossh. Personally however, I'd use a systemd unit to achieve that outcome. This might be a good start: gist.github.com/drmalex07/c0f9304deea566842490 – Rory Browne Apr 9 at 18:18
  • This is very cool, I would definitely go this route if you can @JJAbrams – Natolio Apr 9 at 18:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.