How do you verify if an INCOMING ssh tunnel is still up and who it belongs to?

Question

See edit below for short version

I am having issues displaying the status of the ssh tunnels on our server.

Here is some relevant info:

user001 = remote client's username
123.123.123.123 = ssh server's ip
111.222.111.222 = database server

Right now I'm testing with only one remote client connected. I can see the two reverse tunnels it requested (filtered output):

lsof -i -P
sshd    14652 user001    8u  IPv4 1171285      0t0  TCP localhost:2000 (LISTEN)
sshd    14652 user001    9u  IPv4 1171287      0t0  TCP my.domain.com:2001 (LISTEN)

netstat -atpe --numeric-ports
tcp        0      0 127.0.0.1:2000          0.0.0.0:*               LISTEN      user001    1171285     14652/sshd: user001
tcp        0      0 123.123.123.123:2001     0.0.0.0:*               LISTEN      user001    1171287     14652/sshd: user001

Its very clear here they were initiated by user001 but the regular (-L) tunnel it creates is not listed by lsof. The only info I can find that relates to it is displayed by netsat and it doesn't even seem to be the tunnel itself but rather the network connection between the ssh server and the database server:

tcp        0      0 123.123.123.123:60299    111.222.111.222:52849     TIME_WAIT   root       0           -               
tcp        0      0 123.123.123.123:60300    111.222.111.222:52849     TIME_WAIT   root       0           -               

Which is a pita because not only is it showing under the root user, it only shows local IPs so I have no way of knowing who's tunnel/connection it is. When all the 50+ remote clients are connected its going to be a total mess to troubleshoot.

The Questions:

Is there a better way to display ssh tunnels? Better command? Some program?

Is there any way to have this show under the remote client's user instead or at least show the remote client's ip?

Why is lsof not displaying it?

Thank you.

-----EDIT-----

Ok if you want scratch all that, here is a simplified question:

SSH CLIENT connects to SSH SERVER and requests a tunnel ( using -L ) from his local port 100 to port 5000 on a Database Server on the same network as SSH SERVER.

So we get a tunnel that looks like this:

CLIENT:100 -----> SSH SERVER -----> DBSERVER:5000

How do you verify if this tunnel is up when you are logged in the SSH SERVER ?

I do not want to see if the SSH SERVER is communicating to the Database Server I want to see if the tunnel initiated by SSH CLIENT is still active.

The info displayed by netstat only shows that the server has an active connection from a random port to the port 5000 on the ssh server, it doesn't say why this connection is up (no pid linked to it) and it shows root as the owner (which is completely useless).

Answer 1

I'm pretty sure that the 2 connections in TIME_WAIT are from the server end of the connection, and if the servers are running as root, that makes sense.

Note that the -L option doesn't create the actual tunnel, just the tunnel functionality. The act of connecting to the input IP/Port is what causes SSH to actually create the tunnel to the destination IP/Port; you'll then see the connection in lsof/netstat/etc in an ESTABLISHED state, until someone closes the connection.

Open another terminal window and just type "telnet IP port" to activate the tunnel (this should be the IP/port on the front end of the tunnel you specify with -L), no need to type anything. Then pop over to your server window and run your lsof/netstat commands to see the ESTABLISHED connection in the act.

Hopefully this helps... let me know!