Need further explanation on TIME_WAIT

Question

Hi Unix/Linux Gurus out there!

I need a solid proof that TIME_WAIT (a lot of it actually) is the real culprit in the slowdown in one of our servers. The server is hosted on Parallels Baremetal virtualization, and the actual server is a VM: CentOS5 with dual CPU and 2GB RAM.

A week ago, we started to notice that it was too slow that even doing an 'ls' on a directory with just a few files in there (around 20) would need around 1.5 seconds to display the results.

I tried doing vmstat but it doesn't seem to be even using it's swap. No bottlenecks on the network. But running top, you'd see java mostly hogging the resource. Java is needed since this VM is our hudson server.

One of my colleagues tried checking the connections via

$ vmstat -vatpno

And noticed that there where a lot of connections in TIME_WAIT...around 300+. So we tried applying some of the recommendations in this page particularly that of TCP_FIN_TIMEOUT, TCP_KEEPALIVE_INTERVAL & TCP_KEEPALIVE_PROBES. The connections in TIME_WAIT went lower but still fluctuates between 220 to 280(maybe due to the fact that a new connection is added from time to time and other connections in TIME_WAIT is not yet "timed-out"). Perhaps we could try adding TCP_TW_RECYCLE & TCP_TW_REUSE later when we don't see any improvement.

Now going back to my main question: is there a solid evidence that a lot of TIME_WAIT'ed connections eat up a lot of RAM?

Thanks in advance.

Answer 1

A connection in the TIME_WAIT state is simply waiting to see if any last straggling data packets make their way through the network from the other end, so that they don't get mixed in with another connection's packets. It doesn't actually do anything with those packets. So if anything, a TIME_WAIT connection uses fewer resources than an open connection.

A well-provisioned webserver these days can handle over 10,000 simultaneous connections (note that that was written in 2003, and Moore's Law keeps on marching). Since, if anything, a connection in the TIME_WAIT state will use up less memory than an open connection, 300 connections in TIME_WAIT should be nothing.

For more info on TIME_WAIT, see http://tangentsoft.net/wskfaq/articles/debugging-tcp.html and http://developerweb.net/viewtopic.php?id=2941.

Meanwhile, I wonder how your disk I/O usage looks. Heavy disk I/O can slow down the Linux kernel far more easily than heavy CPU usage, in my experience. You may want to look into the iostat and dstat tools, and see what they tell you.