5

We have a problem with doing SSH connections through remote port forwarding.

The scenario is an enterprise network, where a server on the internal network (let's call it "origin") must log in via SSH to a server in the DMZ ("target"). Since the target server in the DMZ is locked down for connections from the internal network (and cannot even be seen from the internal network), we have a jump host in the DMZ that we go through ("jumphost"). We do this by setting up remote port forwarding on the jumphost.

We run this command from the origin server on the internal network, to the jumphost:

origin> ssh -R *:1234:target:22 myusername@jumphost

This is to establish an SSH session on the jumphost, make it start listening on port 1234 (just an example arbitrary port number), and forward connections on that port to the target server port 22 (SSH).

Then we establish a second SSH session, still from the origin server to the jumphost, on port 1234, which then actually gets connected to the target server on port 22 - this is our 'real' SSH session where we can do our work on the target server:

origin> ssh jumphost -P 1234

Configuration

The jump host has been configured to allow remote port forwarding, with the following settings in sshd_config:

AllowTcpForwarding yes
GatewayPorts yes

Also, firewall openings are in place between the origin server and the jump host, for port 22 (for the initial SSH connection to set up the remote port forwarding) and port 1234 (for the subsequent SSH connection on the forwarded port). There is also a firewall between the jumphost and the target, which has been opened on port 22.

Outcome

When we establish the second connection (the one through the forwarded port), the connection is immediately closed ('connection closed by remote host').

Running tcpdump on the target server shows no activity, i.e. it seems the connection gets blocked.

However, we are able to successfully establish a regular SSH session from the jumphost to the target. Only when coming in through the forwarded port is the connection closed, although both connect to the target on port 22.

What's more, if we make the port forwarding point to a server on the internal network (i.e. a connection from origin on the internal network, to the jumphost in the DMZ, and back to a third server on the internal network), then the SSH session is established successfully.

Speculation and questions

All this leads me to believe that some network security setting is at play, which prevents connecting via the forwarded port on the jump server to the target server within the DMZ. Unfortunately I am not knowledgeable enough to know:

(1) Is an SSH connection coming from the origin server, through a forwarded port on the jump server, 'different', from a network security policy point of view, that it could technically be blocked, and if so, how? And what would need to be done to lift that restriction?

(2) Any other reasons that this connection is not allowed through - firewall configuration, router configuration, SSH settings on the origin or jumphost, anything else?

(3) Could it fail because the origin server does not know the target server, and thus the first ssh command does not work as intended? In other words, is the hostname specified in the first ssh command ("target") interpreted on the client (origin) or on the server we are connecting to to create the tunnel (the jumphost)?

What stumps me the most is that a regular SSH session can be established from the jumphost to the target, I would think that the SSH connection coming in over the forwarded port would be the same, but somehow it isn't.

Any input very much appreciated.

  • Are you absolutely certain that you want to use remote port-forwarding on the origin host? I might be confused here, but are you sure you don't want local port-forwarding instead? This diagram might help with communication: dirk-loss.de/ssh-port-forwarding.htm – igal Dec 1 '17 at 23:31
  • Yes you are right, it looks like I mixed up remote and local ports (it is a bit confusing/backwards, as your link describes). I am going to try this with the local ports. – johnyzee Dec 2 '17 at 21:36
  • Ok. That's what I thought. I'm going to post a solution. – igal Dec 2 '17 at 21:42
  • I turned my comment into a solution and also posted an explanation of the local-host syntax. Let me know if anything I wrote is incorrect or if anything is missing from the solution. – igal Dec 2 '17 at 22:09
  • Did this solve your issue, or is there still a problem? – igal Dec 3 '17 at 15:09
6

It looks like you should be using local port-forwarding instead of remote port-forwarding. You may want to refer to the following helpful blog post by Dirk Loss:

It includes the following illustrative diagram:

ssh port forwarding: local vs remote

In order to read the diagram you need to know that it describes the relationships between 4 different roles involved in creating and utilizing an SSH tunnel:

  • an ssh client (i.e. ssh - the OpenSSH command-line client) used to establish the tunnel;
  • an ssh server (i.e. sshd - the OpenSSH server daemon) used to maintain the other end of the tunnel;
  • an application server (e.g. another ssh server or an http server);
  • an application client (e.g. another ssh client or a web-browser) which wants to access the application server via the tunnel.

It's also important to understand that the two different types of forwarding correspond to two different use cases:

  • Local Forwarding: where the application client connects via the ssh client

  • Remote Forwarding: where the application client connects via the ssh server

Remote forwarding is so called because the forwarding is performed remotely (at the ssh server) rather than locally (at the ssh client). I also find "remote forwarding = reverse forwarding" to be a useful mnemonic.

So as you can see, in order to initiate a connection from an ssh client at the origin host through an sshd server on a proxy jumphost to a third target host you would have to use local port-forwarding. Remote port-forwarding is for the case in which you want the entry point to the tunnel to be located at the host running the sshd server rather than at the host running the ssh client.

In the man page the local port-forwarding syntax is written as follows:

ssh -L [bind_address:]port:host:hostport user@remote

This can be written more intuitively as the following:

ssh -L [local_bind_address:]local_port:remote_host:remote_host_port user@proxy_host

Or, using your naming conventions:

ssh -L [origin_bind_address:]origin_port:target_host:target_host_port user@jump_host

If we modify your command to use local port-forwarding instead then we end up with the following:

user@origin:~$ ssh -L *:1234:target:22 myusername@jumphost
  • 1
    It works great with the local port forwarding. Thanks for the exhaustive answer. I have to admit that the diagrams are still rather confusing in my opinion. – johnyzee Dec 4 '17 at 12:38
  • @johnyzee Yeah, I agree. I think that the diagrams by themselves aren't completely intuitive. I had to read the entire post and think about for a good bit before appreciating the diagrams. It helps to keep the 4 roles in mind: application client, application server, ssh client and ssh server. Then it becomes a question of whether the application client connects via the ssh client or via the ssh server. Connecting via the ssh client is the local port-forwarding scenario and connecting via the server is the remote port-forwarding scenario. A good mnemonic might be "remote=reverse". – igal Dec 4 '17 at 12:42
  • 1
    Yes, "reverse" makes it a lot more clear what it does. – johnyzee Dec 4 '17 at 12:52
  • @johnyzee I updated my solution to include a more detailed explanation of how to interpret the diagram. Hopefully that makes it clearer what's going on (for anyone else who might come along). I definitely spent a fair amount of time being confused about local-vs-remote ssh tunneling before I came across that post - defintely a eureka moment for me. – igal Dec 4 '17 at 12:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.