19

If I run something like this:

ssh -4 -f -N -T -R "/home/dude/lol.socket:192.168.4.44:4444" dude@someserver -p 22 -i privatekey -o "ExitOnForwardFailure yes" -o ConnectTimeout=5 -o ConnectionAttempts=3 -o ServerAliveInterval=15 -o

And then lets say the connection is closed or dies for whatever reason.. say the computer reboots due to maintenance or error or there's internet connectivity issues or whatever -> we have a big problem. The created socket file /home/dude/lol.socket on the someserver does not get deleted by sshd. So as the reverse tunnel initiator is recovering and tries to recreate the tunnel it can't because:

Error: remote port forwarding failed for listen path /home/dude/lol.socket

On the server side you get something like:

error: bind: Address already in use
error: unix_listener: cannot bind to path: /home/dude/lol.socket

What would be the supported way / best hack to cleanup the socket after disconnect? Is this a bug in sshd, shouldn't it do that automatically if/when disconnects are noticed?

Backstory:

The idea behind using the sockets is simply that the server is going to handle n "dudes" creating reverse tunnels for m "lol" services in whatever ports and using sockets makes it much easier to ensure that a "dude" can only access and bind to his own sockets, but not other dudes sockets. It also frees me from having to keep record of which dude is using which port to expose which service. And when dude wants to connect to the service on some other server all he needs to know is the name of the service and bind it to some random local port (or socket if he wants to) i.e.

ssh -v -i -4 -N -T -L "127.0.0.1:3334:/home/dude/lol.sock" -p 22 dude@someserver -o "ExitOnForwardFailure yes" -o ConnectTimeout=5 -o ConnectionAttempts=3 -o ServerAliveInterval=15 -o ServerAliveCountMax=3

There's no need to know some magic port number that the reverse tunnel on the server is suppose to be running on. So, if you have better ideas how to solve this issue I'm all ears.

Tested with client/server both running Debian 9 (client actually on mac inside docker container) using openssh-client/server version 7.4p1-10+deb9u2

5
  • Have a look at autossh Feb 28, 2018 at 12:26
  • 2
    @RuiFRibeiro can you elaborate on how would autossh solve my problem with the sockets on the server not being cleared? I thought it's basically a client side wrapper around ssh to detect broken tunnels and recreate them?
    – Timo
    Feb 28, 2018 at 13:04
  • Your system might be providing each user with their own directory under /run/user/1000/ and so on you could use instead. This is a tmpfs lost on reboot.
    – meuh
    Mar 1, 2018 at 19:41
  • Maybe that is the intended way of using reverse tunnels? Does not suit our purposes though as the server can't be constantly rebooting and disconnecting everyone. But then again bound ports are freed after tunnels get closed, why aren't socket addresses?
    – Timo
    Mar 1, 2018 at 19:56
  • @meuh Right now my best hack to solve this is to basically netstat -a --unix and cross reference that to all socket files I can find based on our naming convention and delete all socket files that weren't listed by netstat. This is now run on cron every minute. It seems to be working alright, maybe I'll post it as an answer at some point if better candidates do not appear. Probly should move the files under /run anyway though as putting them under home directories is rather unorthodox. So thanks @meuh for reminding about that. :)
    – Timo
    Mar 1, 2018 at 20:13

1 Answer 1

33

TL;DR;

The solution is to set the value of StreamLocalBindUnlink to yes in sshd configuration on the server: sudo sh -c 'echo "StreamLocalBindUnlink yes" >> /etc/ssh/sshd_config'.

Long story

The reason this happens is because unix socket files are not automatically removed when the socket is closed. They need to be manually cleaned up when closing if this is desired by calling remove/ unlink with the filepath, but openssh does not do this. However, as I researched the subject further I came to the realisation that the "best practice" with unix sockets is to unlink right before binding to it (Check this SO answer for more details). And this is exactly what the StreamLocalBindUnlink yes tells sshd to do.

Man page says:

 StreamLocalBindUnlink
         Specifies whether to remove an existing Unix-domain socket file for local or remote port forwarding before creating a new
         one.  If the socket file already exists and StreamLocalBindUnlink is not enabled, sshd will be unable to forward the port to
         the Unix-domain socket file.  This option is only used for port forwarding to a Unix-domain socket file.

         The argument must be yes or no.  The default is no.

The downside of this approach is that rebinding to the socket is now allowed even if the old connection is still there. Doing this seems to leave the old tunnel hanging in there so that any existing tcp connections going through that remain intact, but all new connections go to the new tunnel. Also the old tunnel seems to be permanently and irreversibly detached from the filesystem socket address and will not be able to receive anymore new connections even if the new tunnel is closed.

References

7
  • 4
    As of OpenSSH_7.6p1 (and probably earlier) you don't need to modify the server configuration; you can specify -o StreamLocalBindUnlink=yes on your SSH client command line.
    – cjs
    Sep 26, 2018 at 3:06
  • 1
    "Clients!", he shrieked in disgust. "I wouldn't trust any of them to sit right way on a toilet seat."
    – Timo
    Aug 27, 2020 at 12:18
  • Note that StreamLocalBindUnlink might be bugged and not work as advertised. I haven't tried it (yet).
    – Thomas
    Oct 29, 2020 at 15:15
  • So @Thomas if I understood the issue raised in the bug report properly the client side option cjs is advertising is not actually working as expected. Using the server side configs (as I did) should however work quite reliably, right?
    – Timo
    Nov 2, 2020 at 14:42
  • 1
    You need sudo sh -c 'echo "StreamLocalBindUnlink yes" >> /etc/ssh/sshd_config' to get the output redirection to be blessed, otherwise you can't modify the file, even though root ran the echo command.
    – kbro
    Jul 23, 2021 at 9:01

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .