SSH session through jumphost via remote port forwarding

Question

We have a problem with doing SSH connections through remote port forwarding.

The scenario is an enterprise network, where a server on the internal network (let's call it "origin") must log in via SSH to a server in the DMZ ("target"). Since the target server in the DMZ is locked down for connections from the internal network (and cannot even be seen from the internal network), we have a jump host in the DMZ that we go through ("jumphost"). We do this by setting up remote port forwarding on the jumphost.

We run this command from the origin server on the internal network, to the jumphost:

origin> ssh -R *:1234:target:22 myusername@jumphost

This is to establish an SSH session on the jumphost, make it start listening on port 1234 (just an example arbitrary port number), and forward connections on that port to the target server port 22 (SSH).

Then we establish a second SSH session, still from the origin server to the jumphost, on port 1234, which then actually gets connected to the target server on port 22 - this is our 'real' SSH session where we can do our work on the target server:

origin> ssh jumphost -P 1234

Configuration

The jump host has been configured to allow remote port forwarding, with the following settings in sshd_config:

AllowTcpForwarding yes
GatewayPorts yes

Also, firewall openings are in place between the origin server and the jump host, for port 22 (for the initial SSH connection to set up the remote port forwarding) and port 1234 (for the subsequent SSH connection on the forwarded port). There is also a firewall between the jumphost and the target, which has been opened on port 22.

Outcome

When we establish the second connection (the one through the forwarded port), the connection is immediately closed ('connection closed by remote host').

Running tcpdump on the target server shows no activity, i.e. it seems the connection gets blocked.

However, we are able to successfully establish a regular SSH session from the jumphost to the target. Only when coming in through the forwarded port is the connection closed, although both connect to the target on port 22.

What's more, if we make the port forwarding point to a server on the internal network (i.e. a connection from origin on the internal network, to the jumphost in the DMZ, and back to a third server on the internal network), then the SSH session is established successfully.

Speculation and questions

All this leads me to believe that some network security setting is at play, which prevents connecting via the forwarded port on the jump server to the target server within the DMZ. Unfortunately I am not knowledgeable enough to know:

(1) Is an SSH connection coming from the origin server, through a forwarded port on the jump server, 'different', from a network security policy point of view, that it could technically be blocked, and if so, how? And what would need to be done to lift that restriction?

(2) Any other reasons that this connection is not allowed through - firewall configuration, router configuration, SSH settings on the origin or jumphost, anything else?

(3) Could it fail because the origin server does not know the target server, and thus the first ssh command does not work as intended? In other words, is the hostname specified in the first ssh command ("target") interpreted on the client (origin) or on the server we are connecting to to create the tunnel (the jumphost)?

What stumps me the most is that a regular SSH session can be established from the jumphost to the target, I would think that the SSH connection coming in over the forwarded port would be the same, but somehow it isn't.

Any input very much appreciated.

Answer 1

It looks like you should be using local port-forwarding instead of remote port-forwarding. You may want to refer to the following helpful blog post by Dirk Loss:

• SSH port forwarding visualized

It includes the following illustrative diagram:

In order to read the diagram you need to know that it describes the relationships between 4 different roles involved in creating and utilizing an SSH tunnel:

• an ssh client (i.e. ssh - the OpenSSH command-line client) used to establish the tunnel;
• an ssh server (i.e. sshd - the OpenSSH server daemon) used to maintain the other end of the tunnel;
• an application server (e.g. another ssh server or an http server);
• an application client (e.g. another ssh client or a web-browser) which wants to access the application server via the tunnel.

It's also important to understand that the two different types of forwarding correspond to two different use cases:

• Local Forwarding: where the application client connects via the ssh client

• Remote Forwarding: where the application client connects via the ssh server

Remote forwarding is so called because the forwarding is performed remotely (at the ssh server) rather than locally (at the ssh client). I also find "remote forwarding = reverse forwarding" to be a useful mnemonic.

So as you can see, in order to initiate a connection from an ssh client at the origin host through an sshd server on a proxy jumphost to a third target host you would have to use local port-forwarding. Remote port-forwarding is for the case in which you want the entry point to the tunnel to be located at the host running the sshd server rather than at the host running the ssh client.

In the man page the local port-forwarding syntax is written as follows:

ssh -L [bind_address:]port:host:hostport user@remote

This can be written more intuitively as the following:

ssh -L [local_bind_address:]local_port:remote_host:remote_host_port user@proxy_host

Or, using your naming conventions:

ssh -L [origin_bind_address:]origin_port:target_host:target_host_port user@jump_host

If we modify your command to use local port-forwarding instead then we end up with the following:

user@origin:~$ ssh -L *:1234:target:22 myusername@jumphost