SSH fork kills connection

Question

I am using a Linux script which has the task of forwarding control of the system to remote support. In this script one of the commands is a ssh port forward command that will forward the port of the Video Live Stream of a remote camera. On the system with the remote camera, that system is an unknown and thus assumed always behind a firewall and also has a user whom lacks the knowledge to port forward their router and also acquire a dynamic DNS. To overcome this the "CLIENT" system or the camera computer executes the command below:

ssh -R 8089:dc-bb7925e7:8085 -p 2250 [email protected] -fNT

which is forwarding the CLIENT port for the camera feed 8085 to the remote support server 8089. Remote support is supposed to be able to go to localhost:8089 and be able to view the live stream. The problem is that this does not work. Once I insert the -f flag into the command, this command breaks and forwards nothing.

Regardless of the flag, the problem is that when this ssh command executes, all other scripts and processes which are supposed to be running, get put on hold because of the TTY which does not allow the script to exit until the connection is broken. So I tried using the -f to fork the ssh into the background. This does not work as the port does not get forwarded. I can not figure out why.

What I need is for the port to be forwarded and then forgotten about while the connection remains open. It is important that remote support has control over ssh while the client system still operates normally. What am I doing wrong?

If is do not use the -fNT then this functions normally, only all other scripts are not executed.

This is a Debian system.

Answer 1

I don't think the -f is actually causing your problem. I don't see anything wrong in your command, given the right circumstances, your command would work just fine.

However if there is any stateful middlebox on the path between ssh client and ssh server, then if that middlebox loses state your ssh connection will die or stall. That means you need to take extra care, if there is a NAT or a firewall on the path.

Keepalive messages can prevent the connection from dying due to a timeout. They won't prevent the connection from dying due to other reasons such as a middlebox being restarted, but in those cases a keepalive can detect that it has happened and ensure the connection dies right away instead of stalling.

On the client you can use -o ServerAliveInterval=299 or something similar to send keepalives and ensure that the ssh command will terminate soon in case the connection stalls. By wrapping that in a looping shell script, you can ensure it will keep opening a new ssh connection each time the previous connection dies (but obviously that would require the script to fork rather than the ssh command).

If you wanted to keep a port forwarding opened with -L alive, then ServerAliveInterval and respawning the ssh command would be sufficient. But with -R there is another thing, which can go wrong.

If anything is already listening on the port on the server, then -R will fail to set up a port forwarding, but the ssh command will keep running anyway. Thus a loop intended to respawn the ssh command would do no good in that case.

The fix for that is to pass -o ExitOnForwardFailure=yes to the ssh command. Obviously if you just start ssh in a loop with that combination of flags, it could try many times in a row to bind to the same occupied port. So a sleep between successive attempts would be a very good idea. Each time the ssh command dies, your script should simply sleep for a few seconds before trying again.

I would be surprised if there didn't already exist multiple scripts which does everything I have described above.

What I mentioned so far doesn't cover everything you need.

If the connection stalls and the client notice and try to respawn the ssh command, the server might not yet have noticed, thus the port could still be occupied on the server side. In case somebody had configured a firewall in a stupid way, that problem could persist indefinitely with the script on the client trying repeatedly to bind to a port, which remains blocked.

Since convincing everybody to fix their firewall configurations is unlikely to turn out well, you are better off avoiding that problem with an adjustment to your sshd_config, the ClientAliveInterval setting can be used to cause the server to send keepalive messages to the client, and close the connection if it does not receive a reply.

Combining all of the above should be sufficient to keep the port forwarding alive at all times (assuming there is network connectivity between the client and the server).