How to cleanup SSH reverse tunnel socket after connection closed?

Question

If I run something like this:

ssh -4 -f -N -T -R "/home/dude/lol.socket:192.168.4.44:4444" dude@someserver -p 22 -i privatekey -o "ExitOnForwardFailure yes" -o ConnectTimeout=5 -o ConnectionAttempts=3 -o ServerAliveInterval=15 -o

And then lets say the connection is closed or dies for whatever reason.. say the computer reboots due to maintenance or error or there's internet connectivity issues or whatever -> we have a big problem. The created socket file /home/dude/lol.socket on the someserver does not get deleted by sshd. So as the reverse tunnel initiator is recovering and tries to recreate the tunnel it can't because:

Error: remote port forwarding failed for listen path /home/dude/lol.socket

On the server side you get something like:

error: bind: Address already in use
error: unix_listener: cannot bind to path: /home/dude/lol.socket

What would be the supported way / best hack to cleanup the socket after disconnect? Is this a bug in sshd, shouldn't it do that automatically if/when disconnects are noticed?

Backstory:

The idea behind using the sockets is simply that the server is going to handle n "dudes" creating reverse tunnels for m "lol" services in whatever ports and using sockets makes it much easier to ensure that a "dude" can only access and bind to his own sockets, but not other dudes sockets. It also frees me from having to keep record of which dude is using which port to expose which service. And when dude wants to connect to the service on some other server all he needs to know is the name of the service and bind it to some random local port (or socket if he wants to) i.e.

ssh -v -i -4 -N -T -L "127.0.0.1:3334:/home/dude/lol.sock" -p 22 dude@someserver -o "ExitOnForwardFailure yes" -o ConnectTimeout=5 -o ConnectionAttempts=3 -o ServerAliveInterval=15 -o ServerAliveCountMax=3

There's no need to know some magic port number that the reverse tunnel on the server is suppose to be running on. So, if you have better ideas how to solve this issue I'm all ears.

Tested with client/server both running Debian 9 (client actually on mac inside docker container) using openssh-client/server version 7.4p1-10+deb9u2

Answer 1

TL;DR;

The solution is to set the value of StreamLocalBindUnlink to yes in sshd configuration on the server: sudo sh -c 'echo "StreamLocalBindUnlink yes" >> /etc/ssh/sshd_config'.

Long story

The reason this happens is because unix socket files are not automatically removed when the socket is closed. They need to be manually cleaned up when closing if this is desired by calling remove/ unlink with the filepath, but openssh does not do this. However, as I researched the subject further I came to the realisation that the "best practice" with unix sockets is to unlink right before binding to it (Check this SO answer for more details). And this is exactly what the StreamLocalBindUnlink yes tells sshd to do.

Man page says:

 StreamLocalBindUnlink
         Specifies whether to remove an existing Unix-domain socket file for local or remote port forwarding before creating a new
         one.  If the socket file already exists and StreamLocalBindUnlink is not enabled, sshd will be unable to forward the port to
         the Unix-domain socket file.  This option is only used for port forwarding to a Unix-domain socket file.

         The argument must be yes or no.  The default is no.

The downside of this approach is that rebinding to the socket is now allowed even if the old connection is still there. Doing this seems to leave the old tunnel hanging in there so that any existing tcp connections going through that remain intact, but all new connections go to the new tunnel. Also the old tunnel seems to be permanently and irreversibly detached from the filesystem socket address and will not be able to receive anymore new connections even if the new tunnel is closed.

References

• SO: How can I delete a domain socket file when I exit my application
• Discussion about this topic on the openssl-dev mailing list