5

I have a server process and a client process running on the same Linux machine.

Sometimes when I kill -9 the client, I see with tcpdump that a FIN, ACK message is sent. Of course the dead client couldn't have done it because he's dead brutally with SIGKILL. So I guess Linux OS handles the connection closing.

Sometimes I don't see any connection-close-handling and the connection stays “ESTABLISHED” (by netstat).

I always see a connection being closed in Linux ubuntu 4.4.0-53-generic.
Sometimes I see a connection being closed in Linux 3.13.11 (pure kernel, not Ubuntu).

My questions are:

  1. Does Linux handles closing connections?
    1.1 In SIGKILL?
    1.2. When application closed properly, but doesn't call close()?
  2. Did this functionality change between kernel versions 3.13.11 and 4.4.0 ? Does Ubuntu has anything with it?
  3. What if those two processes weren't on the same Linux machine: would it behave the same?
  4. Why does the connection sometimes stay “ESTABLISHED”?
  5. I'm aware of TCP keepalive socket options. If Linux really handles closing the connection. Why do they exist? Only if the FIN, ACK packet drops?
  • A clarification, when you mention netstat showing ESTABLISHED after killing the client, is that netstat executed on the client host or the server? – datUser Aug 16 '17 at 21:33
  • 1
    @datUser: The question says that the client host and the server host are the same machine. – G-Man Aug 16 '17 at 21:51
  • @G-Man ah missed that point. Thanks. – datUser Aug 16 '17 at 21:51
  • 1
    When a process terminates because of a signal, internally (under the hood) it calls exit().  And exit() internally calls close() on all open file descriptors. So the behavior with close() should be a subset of the behavior with exit(), which, in turn, should be a subset of the behavior with kill. (And, conversely, the signal-handling code and the exit() function don’t know anything about the networking subsystem, so they won’t do anything to an open connection except close the file descriptor.) I can’t explain why you’re seeing inconsistent behavior. – G-Man Aug 16 '17 at 21:59
  • 1
    A related question is unix.stackexchange.com/questions/386464 . I suspect that when the questioner tells the world where udhcpc and the unnamed service client are in the process tree, and what actually opened the client socket and spawned udhcpc, everything will become clear. – JdeBP Aug 17 '17 at 5:25
2

A broad question. Perhaps someone can weigh in on your question about the kernel TCP stack between specific kernel versions.

A couple of general answers:

From the client side

  1. In the event of a SIGKILL signal, the kernel terminates program execution, and among other things, closes the process's open file descriptors. TCP sockets are handled a bit differently by the kernel than regular files, in that they need to be flushed and go through the TCP shudown process.

    The difference in an immediate 'FIN, ACK' from the client and a longer socket shut down could depend on what state the client TCP connection was in when the client application was terminated. But generally the kernel will close the application's open sockets.

From the server side

  1. A server does not always know when a client disconnects. The most reliable way to determine if a client has hung up is to attempt a read from the socket which returns an EOF.

    TCP is designed to be resilient to connection latency and intermittent failures. This also translates to some challenges reliably detecting disconnections in the event the FIN, ACK 4 way disconnection handshaking does not occur.

Summary

What you might be seeing upon a kill -9 of the client is the server going into the CLOSE_WAIT of the TCP state, where it is waiting for a TCP timeout. This can take a while. Even though the client is gone, if the kernel did not handle the TCP disconnection handshaking, the server will have to timeout.

If I remember this can take several additional seconds and is likely why you still see ESTABLISHED due to running both the client and server on the same host.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.