Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I've got a SLES machine that accumulates TCP connections in a CLOSE_WAIT state for what appears to be forever. These descriptors eventually suck up all available memory. At the moment, I've got 3037 of them, but it was much higher before a hurry-up reboot recently.

What's interesting is that they're not from connections to local ports that I expect to have listening processes. They have no associated PIDs, and their timers seem to have expired.

# netstat -ton | grep CLOSE_WAIT
tcp      176      0 10.0.0.60:54882     10.0.0.12:31663      CLOSE_WAIT  off (0.00/0/0)
tcp       54      0 10.0.0.60:60957     10.0.0.12:4503       CLOSE_WAIT  off (0.00/0/0)
tcp       89      0 10.0.0.60:50959     10.0.0.12:3518       CLOSE_WAIT  off (0.00/0/0)

# netstat -tonp | grep CLOSE_WAIT
tcp       89      0 10.0.0.59:45598     10.0.0.12:1998       CLOSE_WAIT  -                   
tcp       15      0 10.0.0.59:60861     10.0.0.12:1938       CLOSE_WAIT  -                   
tcp        5      0 10.0.0.59:56173     10.0.0.12:1700       CLOSE_WAIT  -     

I'm not a black-belt when it comes to the TCP stack, or kernel networking, but the TCP config seems sane, since these values are default, per the man page:

# cat /proc/sys/net/ipv4/tcp_fin_timeout 
60
# cat /proc/sys/net/ipv4/tcp_keepalive_time 
7200

So what gives? If the timers have expired, shouldn't the stack automatically clear this stuff out? I'm effectively giving myself a long-term DoS as these things build up.

share|improve this question
    
Oh, and my research shows that others are seeing artifacts like this in 'lsof -i'. I'm not seeing anything odd there. –  pboin Mar 25 '11 at 17:43
1  
Try sudo netstat -tonp to see which program this is occurring with. –  BillThor Mar 25 '11 at 21:13
    
The post and my answer stackoverflow.com/a/17697733/540323 will help. –  Amil Waduwawara Jul 17 '13 at 10:54

2 Answers 2

No, there is no timeout for CLOSE_WAIT. I think that's what the off means in your output.

To get out of CLOSE_WAIT, the application has to close the socket explicitly (or exit).

See How to break CLOSE_WAIT.

If netstat is showing - in the process column:

  • are you running with the appropriate privileges and capabilities (e.g. as root)?
  • they could be kernel processes (e.g. nfsd)
share|improve this answer
    
When doing the netstats, I had full privs, yes. I'll go check out the kernel processes angle -- that's a good idea. I'm really stumped, because there aren't supposed to be any listening sockets at all, except for two or three well-known privileged ports. Maybe it's a wierd iptables problem. I'll check that out too. –  pboin Mar 26 '11 at 11:33

CLOSE_WAIT indicates that the client is closing the connection but the application hasn't closed it yet, or the client is not. You should identify which program or programs are having this problem. Try using netstat -tonp 2>&1 | grep CLOSE to determine which programs as holding the connections.

If there are no programs listed, then the service is being provided by the kernel. These are likely RPC services such as nfs or rpc.lockd. Listening kernel services can be listed with netstat -lntp 2>&1 | grep -- -.

Unless the RPC services have been bound to fixed ports, they will bind to ephemeral ports as your connections appear to show. You may also want to check the processes and mounts on the other server.

You can may be able to bind your NFS services to fixed ports by doing the following:

  1. Select four unused ports for NFS (32763-32766 used here)
  2. Add fixed ports for NFS to /etc/services
    rpc.statd-bc    32763/udp                       # RCP statd broadcast
    rpc.statd-bc    32763/tcp
    rpc.statd       32764/udp                       # RCP statd listen
    rpc.statd       32764/tcp
    rpc.mountd      32765/udp                       # RPC mountd
    rpc.mountd      32765/tcp
    rpc.lockd       32766/udp                       # RPC lockd/nlockmgr
    rpc.lockd       32766/tcp
  3. Configure statd to use the options --port 32763 --outgoing-port 32764
  4. Configure rpcmountd to use the option --port 32765
  5. Shutdown and restart NFS and RPC services.
share|improve this answer
    
I wrote that there were no PIDs, but didn't show my work. I made a quick edit per your suggestion, thanks. –  pboin Mar 26 '11 at 2:06
    
@opboin: Added comments on ports without PIDS (kernel services). –  BillThor Mar 26 '11 at 5:42

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.