Orphaned connections in CLOSE_WAIT state

Question

I've got a SLES machine that accumulates TCP connections in a CLOSE_WAIT state for what appears to be forever. These descriptors eventually suck up all available memory. At the moment, I've got 3037 of them, but it was much higher before a hurry-up reboot recently.

What's interesting is that they're not from connections to local ports that I expect to have listening processes. They have no associated PIDs, and their timers seem to have expired.

# netstat -ton | grep CLOSE_WAIT
tcp      176      0 10.0.0.60:54882     10.0.0.12:31663      CLOSE_WAIT  off (0.00/0/0)
tcp       54      0 10.0.0.60:60957     10.0.0.12:4503       CLOSE_WAIT  off (0.00/0/0)
tcp       89      0 10.0.0.60:50959     10.0.0.12:3518       CLOSE_WAIT  off (0.00/0/0)

# netstat -tonp | grep CLOSE_WAIT
tcp       89      0 10.0.0.59:45598     10.0.0.12:1998       CLOSE_WAIT  -                   
tcp       15      0 10.0.0.59:60861     10.0.0.12:1938       CLOSE_WAIT  -                   
tcp        5      0 10.0.0.59:56173     10.0.0.12:1700       CLOSE_WAIT  -     

I'm not a black-belt when it comes to the TCP stack, or kernel networking, but the TCP config seems sane, since these values are default, per the man page:

# cat /proc/sys/net/ipv4/tcp_fin_timeout 
60
# cat /proc/sys/net/ipv4/tcp_keepalive_time 
7200

So what gives? If the timers have expired, shouldn't the stack automatically clear this stuff out? I'm effectively giving myself a long-term DoS as these things build up.

Answer 1

No, there is no timeout for CLOSE_WAIT. I think that's what the off means in your output.

To get out of CLOSE_WAIT, the application has to close the socket explicitly (or exit).

See How to break CLOSE_WAIT.

If netstat is showing - in the process column:

• are you running with the appropriate privileges and capabilities (e.g. as root)?
• they could be kernel processes (e.g. nfsd)

Answer 2

CLOSE_WAIT indicates that the client is closing the connection but the application hasn't closed it yet, or the client is not. You should identify which program or programs are having this problem. Try using

netstat -tonp 2>&1 | grep CLOSE

to determine which programs as holding the connections.

If there are no programs listed, then the service is being provided by the kernel. These are likely RPC services such as nfs or rpc.lockd. Listening kernel services can be listed with

netstat -lntp 2>&1 | grep -- -  

Unless the RPC services have been bound to fixed ports, they will bind to ephemeral ports as your connections appear to show. You may also want to check the processes and mounts on the other server.

You may be able to bind your NFS services to fixed ports by doing the following:

1. Select four unused ports for NFS (32763-32766 used here)
2. Add fixed ports for NFS to /etc/services

   rpc.statd-bc    32763/udp                       # RCP statd broadcast
   rpc.statd-bc    32763/tcp
   rpc.statd       32764/udp                       # RCP statd listen
   rpc.statd       32764/tcp
   rpc.mountd      32765/udp                       # RPC mountd
   rpc.mountd      32765/tcp
   rpc.lockd       32766/udp                       # RPC lockd/nlockmgr
   rpc.lockd       32766/tcp

3. Configure statd to use the options --port 32763 --outgoing-port 32764
4. Configure rpcmountd to use the option --port 32765
5. Shutdown and restart NFS and RPC services.

Answer 3

While the answer of Mikel is likely also correct that kernel-side sockets like NFS can cause this, CLOSE_WAIT - entries without associated FDs are more often:

Connections that were closed by the other side while they were in the local listen() queue before accept() could be called on them, see my detailed answer with repro here.

in a CLOSE_WAIT state for what appears to be forever

They will stay in this state forever until you

• accept() them and then close() them, or
• close() the entire socket (that's probably less of a solution).

This is also explained in detail in the linked answer.

These descriptors eventually suck up all available memory.

That should not happen, because as per man 2 listen, the length of the listen() queue is limited by

/proc/sys/net/ipv4/tcp_max_syn_backlog  # default 4096
/proc/sys/net/core/somaxconn            # default 4096

unless, of course, you have multiple sockets you listen() on, as the above limits are per such socket.