We have a problem with doing SSH connections through remote port forwarding.
The scenario is an enterprise network, where a server on the internal network (let's call it "origin") must log in via SSH to a server in the DMZ ("target"). Since the target server in the DMZ is locked down for connections from the internal network (and cannot even be seen from the internal network), we have a jump host in the DMZ that we go through ("jumphost"). We do this by setting up remote port forwarding on the jumphost.
We run this command from the origin server on the internal network, to the jumphost:
origin> ssh -R *:1234:target:22 myusername@jumphost
This is to establish an SSH session on the jumphost, make it start listening on port 1234 (just an example arbitrary port number), and forward connections on that port to the target server port 22 (SSH).
Then we establish a second SSH session, still from the origin server to the jumphost, on port 1234, which then actually gets connected to the target server on port 22 - this is our 'real' SSH session where we can do our work on the target server:
origin> ssh jumphost -P 1234
The jump host has been configured to allow remote port forwarding, with the following settings in sshd_config:
Also, firewall openings are in place between the origin server and the jump host, for port 22 (for the initial SSH connection to set up the remote port forwarding) and port 1234 (for the subsequent SSH connection on the forwarded port). There is also a firewall between the jumphost and the target, which has been opened on port 22.
When we establish the second connection (the one through the forwarded port), the connection is immediately closed ('connection closed by remote host').
Running tcpdump on the target server shows no activity, i.e. it seems the connection gets blocked.
However, we are able to successfully establish a regular SSH session from the jumphost to the target. Only when coming in through the forwarded port is the connection closed, although both connect to the target on port 22.
What's more, if we make the port forwarding point to a server on the internal network (i.e. a connection from origin on the internal network, to the jumphost in the DMZ, and back to a third server on the internal network), then the SSH session is established successfully.
Speculation and questions
All this leads me to believe that some network security setting is at play, which prevents connecting via the forwarded port on the jump server to the target server within the DMZ. Unfortunately I am not knowledgeable enough to know:
(1) Is an SSH connection coming from the origin server, through a forwarded port on the jump server, 'different', from a network security policy point of view, that it could technically be blocked, and if so, how? And what would need to be done to lift that restriction?
(2) Any other reasons that this connection is not allowed through - firewall configuration, router configuration, SSH settings on the origin or jumphost, anything else?
(3) Could it fail because the origin server does not know the target server, and thus the first ssh command does not work as intended? In other words, is the hostname specified in the first ssh command ("target") interpreted on the client (origin) or on the server we are connecting to to create the tunnel (the jumphost)?
What stumps me the most is that a regular SSH session can be established from the jumphost to the target, I would think that the SSH connection coming in over the forwarded port would be the same, but somehow it isn't.
Any input very much appreciated.