9

If I use a client to create a reverse tunnel with SSH using -R I will get this output on the server if I run netstat:

default@debian:~$ sudo netstat -pln
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      801/sshd
tcp        0      0 127.0.0.1:39963         0.0.0.0:*               LISTEN      1074/sshd: anonymou
tcp6       0      0 :::22                   :::*                    LISTEN      801/sshd
udp        0      0 0.0.0.0:68              0.0.0.0:*                           451/dhclient

I would like to know what is the IP of the client that created the listening socket on port 39963.

What command can I run so I not only get the bound IP:PORT but the IP of the client that created the tunnel?

6
  • We can only see LISTEN-ing sockets on that output. Including that port you are interested on, which is not a connection (yet), it’s just the sshd server itself that has set up the port forwarding you requested with the -R option. To really see the ESTABLISHED connections, take the l (el) away from the netstat command. I suppose you might instead want to use the t option to see only TCP sockets, and also the a option to see both ESTABLISHED connections and LISTEN-ing sockets. That is: netstat -ptan
    – LL3
    Apr 22 '19 at 23:20
  • Hi, Unfortunately that does not give you the source IP for the binded port (127.0.0.1:39963) Apr 22 '19 at 23:53
  • There is no source IP because your server is just waiting for new connections. 127.0.0.1 is the localhost, your server in that case. It is waiting for connections to that port, and only the server machine itself can connect to its own 127.0.0.1. For example, try telnet localhost 39963 from the server: the ssh on your client machine will connect to whatever other host:port you indicated on your ssh -R command. Then you’ll see an ESTABLISHED connection on your server. Only, it won't be interesting because it'll be 127.0.0.1. The real remote IP will be seen from your client
    – LL3
    Apr 23 '19 at 0:22
  • @LL3 I think the OP is looking for the IP of the SSH client connected to the server. In particular he is looking for the SSH client responsible for opening a remote listening socket on the server. There will be a source IP, the challenge is finding it. Apr 23 '19 at 11:33
  • Related (and NOT a duplicate) is unix.stackexchange.com/questions/127940/… Apr 23 '19 at 12:20
6
+50

I can't give you a single command but there are a few commands which will let you find the client. The problem is with the way OpenSSH creates processes. The process which owns the listening socket 39963 is the child of the process which owns the SSH connection. So they will have different PIDs. In your example you need to find the parent process for 1074. The following works for OpenSSH server on Ubuntu. I can't guarantee every distribution will behave the same way.

Start by finding the PID for the client connection. Use grep to filter the output of netstat.

$ sudo netstat -ptln | grep 39963
tcp        0      0 127.0.0.1:39963         0.0.0.0:*               LISTEN      21921/sshd: philip@
tcp6       0      0 ::1:39963               :::*                    LISTEN      21921/sshd: philip@

The listening process here is 21921. You need it's parent:

$ ps -ef | grep 21921
philip   21921 21919  0 11:01 ?        00:00:00 sshd: philip@pts/1
philip   21924 21921  0 11:01 pts/1    00:00:00 -bash
philip   22844 20309  0 11:15 pts/0    00:00:00 grep --color=auto 21921

# Or try this:

$ ps -ef | awk '$2 == 21921 { print $3 }'
21919

The parent of 21921 here is 21919. Now we look back to netstat to find the client:

$ sudo netstat -ptn | grep 21919
tcp        0      0 192.168.1.36:22          192.168.1.10:54425   ESTABLISHED 21919/sshd: philip

This shows the remote ip is 192.168.1.10

2
  • 1
    To get the parent process of $child, consider ps -p "$child" -oppid=
    – Jeff Schaller
    Apr 23 '19 at 14:23
  • I put together an script based on your line of though to automatically detect the port. It's an answer here. Any thoughs on it will be appreciated
    – DGoiko
    Dec 23 '20 at 21:25
3

The SSH client and server exchange messages to set up forwardings, but the kernel network table cares/knows only about its local IP stack, so it sees only sshd.

But, OpenSSH also shares some details in the environment (which too is passed via a message exchange).

 sshcpid=$(pgrep -P 1074)               # fetch child PID
 xargs -0 -L1 -a /proc/$sshcpid/environ # show environment strings

The variables SSH_CLIENT and SSH_CONNECTION should both contain what you need.

Here, xargs is a handy way to process the NUL delimited strings in environ.

This assumes:

  • linux, for a /proc with environ
  • OpenSSH with a process chain like:

    daemon sshd → forked connection sshd → privsep sshd → user shell

e.g. (from pstree -lp):

 init(1)-+
         .
         |-sshd(1015) +-sshd(1072)---sshd(1074)---bash(1075)

The privsep sshd process is the one you see in netstat, the environment you read is from its immediate descendent, the shell (bash in this example).

1

The lsof command in Linux displays in its output information about files that are opened by processes

Taking the process id of ssh from the netstat,

lsof -p $pid -a -d 3

will show IP of other end of the connection. As ssh can be hopped, this might not be the final location.

$pid is process id of your ssh service

if you don't have lsof you can install it with apt-get install lsof

4
  • Please elaborate: (1) According to the example provided by the OP, should $pid in your command be 801 or 1074? (2) What does -a -d 3 do? and why is it useful here? Apr 23 '19 at 6:34
  • 1.pid is an identification number that is automatically assigned to each process you can get pid of sshd by ps -e | grep sshd Apr 23 '19 at 7:28
  • 2. man lsof will give you more information about options Apr 23 '19 at 7:41
  • I know what PIDs are. My point is the example in question shows there are at least two sshd processes, two PIDs that match your description. Which one to use then? and why? In other words: your answer should explicitly state this must be the PID of sshd that listens on the port being forwarded (39963), not of some other (any) sshd. So 1074 in the OP's example. Apr 23 '19 at 7:41
1

As has been mentioned, you need to wait until a connection has been established before you can see the foreign address (on the remote side) of the connection. You could go about getting this information in two ways:

  • Either sample netstat periodically, and then grep out the relevant IP:PORT. Straightforward, but you wouldn't catch short lived connections.
  • Your other alternative is to setup a dedicated log with iptables and then process this. See e.g. Have iptables run command/script on new connection attempt.

Either method runs asynchronously, and you would need to maintain the connection processing separate from ssh.

1
  • But you can't relate the listening port created by a new process to the stablished connection with the client that uses -R. What sshd does is: Recieves the client connection and maintains it open. Then creates a listening port for new connections. When those arrive, they're piped to the client. There's no way for netstat that I know to perform this task. I posted a way that works in some OSs. It gets the parent process of the listening port, which is the one that can be searched in netstat to detect the IP we're looking for.
    – DGoiko
    Dec 22 '20 at 23:17
1

Based on top answer idea, I wrote a simple script to automate the process. The single argument is the listening port, and it returns the ip and outgoing port of the client.

#!/bin/bash
PORT_TUNEL=$1
PID=`fuser $PORT_TUNEL/tcp 2>/dev/null`
# xargs is just used for trimming
PARENT_PID=`ps -p ${PID:-$$} -o ppid= | xargs`
netstat -ptn |  awk -v ppid="$PARENT_PID/sshd:"  '{if($7==ppid) print $5}'

Usage:

rsship 39963

Sample output:

67.214.221.43:26412

Explanation:

This script asumes you already know which listening port you want to obtain the client's IP. As explained in other answers, it is directly impossible to know that from the connection itself. Instead we need to obtain the process id (pid) that is holding the listening port. We're using fuser in order to do that and storing it in a variable. Since fuser has some output in stderr that would mess up with our code, we redirect it to /dev/null. Now we have the PID that holds the listening port in our $PID variable.

The next step it to get the parent process, which is the process maintaining the SSH connection with your client. It uses ps in order to do that. Xargs is used to trim the output, as it may come with some blankspaces that would mess up with our awk later.

At last, it uses netstat to get a list of all the connections, and awk to display column 5 when column 7 matches the string "$PARENT_PID/sshd:". Note that -v option is used to pass a bash variable to awk in the proper way. I'm not going to extend myself on this, search "passing bash variables to awk script" if you need more insights abouts this. I'm declaring ppid variable, holding "$PARENT_PID/sshd:", which is the text I want to match against the 7th col. We're using this instead of grep because 1150/sshd: would match 150/sshd:. There are multiple ways of doing this, but this awk command was the simplest and most readable I could come up with. This comes after some struggles because greping just $PARENT_PID often matched some random ports on diferent connections giving unexpected results.

I'm aware this could be written in a much more compact way, but I like to keep the scripts readable and self-explanatory in a glipse when it makes sense

Listening ports enumerator

I've a script to list ALL listenings reverse tunnels on my machine and fetch their client's IP and outgoing port:

In my setup I have a special SSH server for this purpose, so all connections come through "sshtunnel" user. You'll have to modify the script if you're using your normal ssh server and come up with another way to get your first list of listening ports. using "grep sshd" will ALMOST do the trick, although it will list your main shh server listening ports too, although in MY particular situation it works because ssh server was bind to *, so the grep 127.0.0.1 would filter it out too:

#!/bin/bash

IP_SCRIPT="/etc/openvpn/rsship"

PORTS=`lsof -i -P -n | grep LISTEN | grep sshtunnel | grep 127.0.0.1 | awk -F" " '{print $9}' | awk -F":" '{print $2}' | awk '{if($1==$1+0 && $1>16000 && $1<16255) print $1}' | sort -n`


while IFS= read -r line; do
    REAL_IP=$($IP_SCRIPT $line)
    echo $line$'\t'$REAL_IP
done <<< "$PORTS"

Note: $IP_SCRIPT references to the first script posted.

grep 127.0.0.1 is used to remove ipv6 lines AND those binded to all interfaces (marked in the output with asterisk *) in a single pass (in our setup, all IPv6 have IPv4 counterparts).

This script only lists ports between 16000 and 16255 (because thats my particular use case). You can remove the whole awk part to make it accept ANY port. Like this:

#!/bin/bash

IP_SCRIPT="/etc/openvpn/rsship"

PORTS=`lsof -i -P -n | grep LISTEN | grep sshtunnel | grep 127.0.0.1 | awk -F" " '{print $9}' | awk -F":" '{print $2}' | sort -n`


while IFS= read -r line; do
    REAL_IP=$($IP_SCRIPT $line)
    echo $line$'\t'$REAL_IP
done <<< "$PORTS"

The lsof part can be made better, though, it was already made, and since I was in a rush I decided to reuse that code without giving it a though. It is basically using awk to split and print a particular text of a column (the port, specifically). As we usually say: works on my machine. If you want to improve it, use "lsof -i -P -n | grep LISTEN" and see how can you narrow it down further it in a better way, as my line may not be suitable for your particular case. I'm not 100% sure, but I think that the same results can be achieved using only netstat.

If you're running this script very frequently and have multiple ports, you may consider giving it a twist so it calls netstat just once and process that output over and over to fetch all data.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.