When/how does Linux decides to close a socket on application kill?

Question

I have a server process and a client process running on the same Linux machine.

Sometimes when I kill -9 the client, I see with tcpdump that a FIN, ACK message is sent. Of course the dead client couldn't have done it because he's dead brutally with SIGKILL. So I guess Linux OS handles the connection closing.

Sometimes I don't see any connection-close-handling and the connection stays “ESTABLISHED” (by netstat).

I always see a connection being closed in Linux ubuntu 4.4.0-53-generic.
Sometimes I see a connection being closed in Linux 3.13.11 (pure kernel, not Ubuntu).

My questions are:

1. Does Linux handles closing connections?
   1.1 In SIGKILL?
   1.2. When application closed properly, but doesn't call close()?
2. Did this functionality change between kernel versions 3.13.11 and 4.4.0 ? Does Ubuntu has anything with it?
3. What if those two processes weren't on the same Linux machine: would it behave the same?
4. Why does the connection sometimes stay “ESTABLISHED”?
5. I'm aware of TCP keepalive socket options. If Linux really handles closing the connection. Why do they exist? Only if the FIN, ACK packet drops?

Answer 1

A broad question. Perhaps someone can weigh in on your question about the kernel TCP stack between specific kernel versions.

A couple of general answers:

From the client side

1. In the event of a SIGKILL signal, the kernel terminates program execution, and among other things, closes the process's open file descriptors. TCP sockets are handled a bit differently by the kernel than regular files, in that they need to be flushed and go through the TCP shudown process.

   The difference in an immediate 'FIN, ACK' from the client and a longer socket shut down could depend on what state the client TCP connection was in when the client application was terminated. But generally the kernel will close the application's open sockets.

From the server side

1. A server does not always know when a client disconnects. The most reliable way to determine if a client has hung up is to attempt a read from the socket which returns an EOF.

   TCP is designed to be resilient to connection latency and intermittent failures. This also translates to some challenges reliably detecting disconnections in the event the FIN, ACK 4 way disconnection handshaking does not occur.

Summary

What you might be seeing upon a kill -9 of the client is the server going into the CLOSE_WAIT of the TCP state, where it is waiting for a TCP timeout. This can take a while. Even though the client is gone, if the kernel did not handle the TCP disconnection handshaking, the server will have to timeout.

If I remember this can take several additional seconds and is likely why you still see ESTABLISHED due to running both the client and server on the same host.