How to continuously monitor PIDs connecting to local sockets?

Question

Some local processes are connecting and disconnecting to 127.0.0.1 on port 1234.

I would like to log all connection to this port (or to server process).

I've tried

 ss -tpn | grep 1234 

It shows a list, but then stops; It doesn't keep logging new connections, so I can't catch the sockets nor PID of owner.

What tool can I use to discover PID of the connecting and disconnecting sockets?

Answer 1

There are two tools you could use to monitor TCP connect events on Linux:

• tcpconnect.py in bcc tools (pkgs.org, repology)
• tcpconnect.bt in bpftrace (pkgs.org, repology)

The difference between the two is that the former provides options for customizing output (e.g., filtering by PID or port number) while the latter is a more simplistic tool and doesn't provide fancy options.

For your use case, the most simple option would be to install bcc and run:

tcpconnect.py -P 1234

If you install these tools using your distro's package manager, keep in mind that some distros don't place tcpconnect in /usr/bin and place them under somewhere else like /usr/share instead. So be sure to check where your distro places these files if you can't find them.

Answer 2

there is a "watch" tool that does that, or you can do
while true; do sleep 2; clear; ss -tulnp | grep 1234 ; done
you can append the code to a file and keep a track of it. which will probably generate cpu usage, but may help you with what you're looking for.
Linux watch command

Answer 3

While ss only dumps socket statistics, and you would need to simulate continuous dumps (with watch or a while loop), netstat has a continuous mode(-c).

netstat -ntcp | grep '  127.0.0.1:1234'

Note the two spaces before 127... is not a typo, if you need to capture only the client pid.
Sample output:

$ netstat -ntcp | grep '  127.0.0.1:1234'
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp        0      0 127.0.0.1:50146         127.0.0.1:1234          ESTABLISHED 21274/nc            
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp        0      0 127.0.0.1:50146         127.0.0.1:1234          ESTABLISHED 21274/nc            
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp        0      0 127.0.0.1:50146         127.0.0.1:1234          ESTABLISHED 21274/nc            
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp        0      0 127.0.0.1:35720         127.0.0.1:1234          ESTABLISHED 22655/nc            
tcp        0      0 127.0.0.1:50146         127.0.0.1:1234          ESTABLISHED 21274/nc            
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp        0      0 127.0.0.1:35720         127.0.0.1:1234          ESTABLISHED 22655/nc            
tcp        0      0 127.0.0.1:50146         127.0.0.1:1234          ESTABLISHED 21274/nc

Answer 4

The ss program is using a sock_diag(7) netlink socket to retrieve information about sockets. But the sock_diag interface doesn't support a "monitor"/watching/listening mode, as rtnetlink(7) does. You can only do queries via a sock_diag socket.

You can however capture the tcp syn (=connect) packets via pcap/tcpdump and use to sock_diag interface to find process info about them.

But there are at least three serious problems with this:

1. all the command line utilities able to query and display that info (ss, lsof) are very rigid, and it's hard to impossible to configure their output format or let them act as a filter. From the command line all you can possibly do is run a separate ss instance for each packet.

2. it's racy; the process may have already finished in the time between capturing the packet and querying information about the process.

3. tcpdump itself is as easy to use from a script as ss or lsof.

Keeping all this in mind, something like this may do, or at least may be a start:

socmon(){
  script /dev/null -qc "tcpdump -qn -iany '
   ((tcp[tcpflags] & tcp-syn) != 0 or (ip6 and (ip6[13 + 40] & 2) == 2)) and ($*)
' 2>/dev/null" </dev/null |
  perl -ne 'print; next unless /(\S+)\.(\d+) >/; print qx(ss -Hp src "[$1]:$2") =~ s/.*users:/\t/r'
}

Usage:

>>> socmon dst port 9999
07:14:59.700995 IP6 fe80::89c8:7f7c:29f5:78df.50720 > fe80::89c8:7f7c:29f5:78df.9999: tcp 0
        (("xxx",pid=15805,fd=1),("xxx",pid=15804,fd=0))
07:15:08.868555 IP 127.0.0.1.42784 > 127.0.0.1.9999: tcp 0
        (("xxx",pid=15851,fd=1),("xxx",pid=15850,fd=0))
07:15:17.055518 IP 172.31.255.1.39700 > 172.31.255.1.9999: tcp 0
        (("xxx",pid=15856,fd=1),("xxx",pid=15855,fd=0))

The script /dev/null ... </dev/null forces tcpdump to use line buffering; tcpdump -l will NOT do without an artificial delay between capturing the packet and printing it.

The (ip6 and (ip6[13 + 40] & 2) == 2) checks if it's a SYN packet BY HAND because tcp[tcpflags] & tcp-syn doesn't work with IPv6.

The perl thing extracts the source address and port from the tcpdump output, calls ss with them as a filter argument, and trims the ss output to show only the process info.

A better idea, not developed here

You could redirect the ports with iptables or nftables to another port, where a listening program could retrieve info about its peer before accepting the connection, and then will forward the connection to the destination. This would fix both the race, and won't be confused by processes which connect somewhere, then either pass or leak the socket file descriptor to other processes.

Answer 5

watch -n 1 -d 'lsof -n -itcp:1234'

not logging, but powerful monitoring in real-time with highlights...

$ man watch

• -n 1 - each 1 second
• -d - Highlight the differences

$ man lsof

• -n - no resolve ips to domains.
• -itcp:1234 - any interacts with tcp port 1234