Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

Hi Unix/Linux Gurus out there!

I need a solid proof that TIME_WAIT (a lot of it actually) is the real culprit in the slowdown in one of our servers. The server is hosted on Parallels Baremetal virtualization, and the actual server is a VM: CentOS5 with dual CPU and 2GB RAM.

A week ago, we started to notice that it was too slow that even doing an 'ls' on a directory with just a few files in there (around 20) would need around 1.5 seconds to display the results.

I tried doing vmstat but it doesn't seem to be even using it's swap. No bottlenecks on the network. But running top, you'd see java mostly hogging the resource. Java is needed since this VM is our hudson server.

One of my colleagues tried checking the connections via

$ vmstat -vatpno

And noticed that there where a lot of connections in TIME_WAIT...around 300+. So we tried applying some of the recommendations in this page particularly that of TCP_FIN_TIMEOUT, TCP_KEEPALIVE_INTERVAL & TCP_KEEPALIVE_PROBES. The connections in TIME_WAIT went lower but still fluctuates between 220 to 280(maybe due to the fact that a new connection is added from time to time and other connections in TIME_WAIT is not yet "timed-out"). Perhaps we could try adding TCP_TW_RECYCLE & TCP_TW_REUSE later when we don't see any improvement.

Now going back to my main question: is there a solid evidence that a lot of TIME_WAIT'ed connections eat up a lot of RAM?

Thanks in advance.

share|improve this question
add comment

1 Answer 1

up vote 2 down vote accepted

A connection in the TIME_WAIT state is simply waiting to see if any last straggling data packets make their way through the network from the other end, so that they don't get mixed in with another connection's packets. It doesn't actually do anything with those packets. So if anything, a TIME_WAIT connection uses fewer resources than an open connection.

A well-provisioned webserver these days can handle over 10,000 simultaneous connections (note that that was written in 2003, and Moore's Law keeps on marching). Since, if anything, a connection in the TIME_WAIT state will use up less memory than an open connection, 300 connections in TIME_WAIT should be nothing.

For more info on TIME_WAIT, see http://tangentsoft.net/wskfaq/articles/debugging-tcp.html and http://developerweb.net/viewtopic.php?id=2941.

Meanwhile, I wonder how your disk I/O usage looks. Heavy disk I/O can slow down the Linux kernel far more easily than heavy CPU usage, in my experience. You may want to look into the iostat and dstat tools, and see what they tell you.

share|improve this answer
    
Thanks for the reply Jander. I could be confident now that the 300+ connections is nothing and could be something else. Perhaps, it could have something to do really with disk or network bottleneck :) –  icasimpan Feb 2 '11 at 5:13
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.