38

I've got a SLES machine that accumulates TCP connections in a CLOSE_WAIT state for what appears to be forever. These descriptors eventually suck up all available memory. At the moment, I've got 3037 of them, but it was much higher before a hurry-up reboot recently.

What's interesting is that they're not from connections to local ports that I expect to have listening processes. They have no associated PIDs, and their timers seem to have expired.

# netstat -ton | grep CLOSE_WAIT
tcp      176      0 10.0.0.60:54882     10.0.0.12:31663      CLOSE_WAIT  off (0.00/0/0)
tcp       54      0 10.0.0.60:60957     10.0.0.12:4503       CLOSE_WAIT  off (0.00/0/0)
tcp       89      0 10.0.0.60:50959     10.0.0.12:3518       CLOSE_WAIT  off (0.00/0/0)

# netstat -tonp | grep CLOSE_WAIT
tcp       89      0 10.0.0.59:45598     10.0.0.12:1998       CLOSE_WAIT  -                   
tcp       15      0 10.0.0.59:60861     10.0.0.12:1938       CLOSE_WAIT  -                   
tcp        5      0 10.0.0.59:56173     10.0.0.12:1700       CLOSE_WAIT  -     

I'm not a black-belt when it comes to the TCP stack, or kernel networking, but the TCP config seems sane, since these values are default, per the man page:

# cat /proc/sys/net/ipv4/tcp_fin_timeout 
60
# cat /proc/sys/net/ipv4/tcp_keepalive_time 
7200

So what gives? If the timers have expired, shouldn't the stack automatically clear this stuff out? I'm effectively giving myself a long-term DoS as these things build up.

4

3 Answers 3

20

No, there is no timeout for CLOSE_WAIT. I think that's what the off means in your output.

To get out of CLOSE_WAIT, the application has to close the socket explicitly (or exit).

See How to break CLOSE_WAIT.

If netstat is showing - in the process column:

  • are you running with the appropriate privileges and capabilities (e.g. as root)?
  • they could be kernel processes (e.g. nfsd)
4
  • When doing the netstats, I had full privs, yes. I'll go check out the kernel processes angle -- that's a good idea. I'm really stumped, because there aren't supposed to be any listening sockets at all, except for two or three well-known privileged ports. Maybe it's a wierd iptables problem. I'll check that out too.
    – pboin
    Commented Mar 26, 2011 at 11:33
  • 1
    The link is broken.
    – Nathan
    Commented May 11, 2015 at 18:33
  • 1
    Thanks, updated to unix.derkeiler.com/Mailing-Lists/SunManagers/2006-01/…
    – Mikel
    Commented May 12, 2015 at 1:45
  • Thanks a lot, this helped me find a bug in my code that was causing CLOSE_WAIT!
    – Hossein
    Commented Apr 24, 2021 at 5:02
11

CLOSE_WAIT indicates that the client is closing the connection but the application hasn't closed it yet, or the client is not. You should identify which program or programs are having this problem. Try using

netstat -tonp 2>&1 | grep CLOSE

to determine which programs as holding the connections.

If there are no programs listed, then the service is being provided by the kernel. These are likely RPC services such as nfs or rpc.lockd. Listening kernel services can be listed with

netstat -lntp 2>&1 | grep -- -  

Unless the RPC services have been bound to fixed ports, they will bind to ephemeral ports as your connections appear to show. You may also want to check the processes and mounts on the other server.

You may be able to bind your NFS services to fixed ports by doing the following:

  1. Select four unused ports for NFS (32763-32766 used here)
  2. Add fixed ports for NFS to /etc/services
    rpc.statd-bc    32763/udp                       # RCP statd broadcast
    rpc.statd-bc    32763/tcp
    rpc.statd       32764/udp                       # RCP statd listen
    rpc.statd       32764/tcp
    rpc.mountd      32765/udp                       # RPC mountd
    rpc.mountd      32765/tcp
    rpc.lockd       32766/udp                       # RPC lockd/nlockmgr
    rpc.lockd       32766/tcp
  3. Configure statd to use the options --port 32763 --outgoing-port 32764
  4. Configure rpcmountd to use the option --port 32765
  5. Shutdown and restart NFS and RPC services.
3
  • I wrote that there were no PIDs, but didn't show my work. I made a quick edit per your suggestion, thanks.
    – pboin
    Commented Mar 26, 2011 at 2:06
  • @opboin: Added comments on ports without PIDS (kernel services).
    – BillThor
    Commented Mar 26, 2011 at 5:42
  • 5
    CLOSE-WAIT means that the peer has closed its end and the local OS is waiting for the local application to close.
    – user207421
    Commented Apr 23, 2017 at 9:31
0

While the answer of Mikel is likely also correct that kernel-side sockets like NFS can cause this, CLOSE_WAIT - entries without associated FDs are more often:

Connections that were closed by the other side while they were in the local listen() queue before accept() could be called on them, see my detailed answer with repro here.

in a CLOSE_WAIT state for what appears to be forever

They will stay in this state forever until you

  • accept() them and then close() them, or
  • close() the entire socket (that's probably less of a solution).

This is also explained in detail in the linked answer.

These descriptors eventually suck up all available memory.

That should not happen, because as per man 2 listen, the length of the listen() queue is limited by

/proc/sys/net/ipv4/tcp_max_syn_backlog  # default 4096
/proc/sys/net/core/somaxconn            # default 4096

unless, of course, you have multiple sockets you listen() on, as the above limits are per such socket.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .