log PID of each connection attempt

Question

Finding the PID of an established connection is trivial using netstat or lsof. However, I have a process which is creating a connection ever 60 seconds to our database and locking it up by maxing out the failed connection attempt limit. I can increase the failed connection limit to something extremely high on the database, or I can try to track down what is making the connection, and I have chosen the latter.

Based on tcpdump/wireshark, I can see that what is happening is that a connection is established and then the connecting server immediately closes the connection before the server can even respond. What I don't know is why.

The first step is to find out what PID is opening the connection. Unfortunately, this seems easier said than done. The problem is that when a connection goes into TIME_WAIT state, it is no longer associated with a PID. Since my connection has a lifetime of less than a tenth of a second, is there any way to record this information?

netstat and lsof appear to be able to poll every second, but this simply isn't fast enough with the connection attempt I am dealing with. Is there a hook that I can connect to to dump this information to a log? Or is my only option to brute force it with a loop and some coding?

I use CentOS 6.

Answer 1

Consider using SystemTap. It is dynamic instrumenting engine that dynamically patches kernel so you can track any in-kernel event such as opening a socket. It is actively developed by RedHat so it is supported in CentOS.

Installing

To install SystemTap on CentOS 6:

1. Enable debuginfo repository:

   sed -i 's/^enabled=0/enabled=1/' /etc/yum.repos.d/CentOS-Debuginfo.repo
   

2. Install SystemTap:

   yum install systemtap
   

3. Install debuginfo packages for kernel. It can be done manually, but there is a tool that can do it automatically:

   stap-prep
   

Tracing

SystemTap has not tapset probe for TCP connection, but you may directly bind to kernel functions! You can also do it on socket level.

I.e. create script called conn.stp:

probe kernel.function("tcp_v4_connect") {
    printf("connect [%s:%d] -> %s:%d\n", execname(), pid(),
            ip_ntop(@cast($uaddr, "struct sockaddr_in")->sin_addr->s_addr),
            ntohs(@cast($uaddr, "struct sockaddr_in")->sin_port));
}

This will give you the following output:

# stap conn.stp
connect [nc:2552] -> 192.168.24.18:50000
connect [nc:2554] -> 192.168.24.18:50000
connect [nc:2556] -> 192.168.24.18:50000

However tracking disconnection events seem to be more trickier.