TCP communication between client and server after server IP has been deleted

Question

I created a tcp server listening at [IP1:PORT1] and connect to it via a client using [IP2]. On both the client and server, SO_KEEPALIVE and TCP_USER_TIMEOUT are enabled with below config values.

TCP_KEEPIDLE = 1
TCP_KEEPINTVL = 1
TCP_KEEPCNT = 4
TCP_USER_TIMEOUT = 5000

In the TCPdump that I captured, I see the following:

1. 3 way handshake happening between client and server in the beginning(syn-synack-ack).
2. Keepalive packets being generated from both client and server after every 1 second, and the respective peer sending the ack in response.

Everything is happening as per my expectations. However now I deleted the server IP(IP1) using the below command:

/sbin/ip addr del IP1 dev DEV

After deleting the IP, this is what I observe in the trace:

1. Keepalive packets continue to be sent from both the entities(client and server). The server is surprisingly still sending keepalive packets, even when the underlying IP has been deleted!

2. The client is responding with acks for the keepalive packets sent by the server.

3. The server is not sending acks for any keepalive packets sent by the client.

4. After about 4 seconds, server IP(IP1) sends [RST,ACK] to the client after which no further packet exchange happens.

So 2 things I didn't understand from above:

1. How come communication continued happening for some time even when the server IP was deleted?
2. Why was the server able to send Keepalive packets to the client but not able to send keepalive acks to the client?

OS: Red Hat Enterprise Linux release 8.3

Answer 1

Very vague answer, but it should put you in the right direction for more investigation:

When you establish a TCP connection, the kernel sets up data structures about this connection. Removing the IP address from the interface doesn't necessarily completely delete this information (because I don't think anyone ever bothered about this case, you'd normally shut down all servers before removing the IP address, and apparently it does get cleaned up eventually).

So these internal data structures apparently enable doing the keepalive exchange, with ACK packets not generated or getting through, until something triggers (probably because the connection is now considered dead) and the whole connection state gets collected.

If you want details, read the kernel code, find out which data structures are established, find out which structures get remove when you delete the IP address, and which stay.

But the whole thing is academic - you found some edge case behaviour that has no practical consequence.