7

I have a hypothesis: sometimes TCP connections arrive faster than my server can accept() them. They queue up until the queue overflows and then there are problems.

How can I confirm this is happening?

Can I monitor the length of the accept queue or the number of overflows? Is there a counter exposed somewhere?

  • You're looking for netstat. – Satō Katsura Dec 7 '16 at 18:26
  • As far as I can tell, netstat only shows the send and receive queue lengths, which is not the same as the accept queue. – Phil Frost Dec 7 '16 at 19:08
  • Yeah, it isn't shown by default. man netstat | less +/Flags – Satō Katsura Dec 7 '16 at 19:33
  • I'm not sure how those flags tell me the accept queue length -- in fact netstat doesn't seem to show Flags at all for TCP connections. From a little testing, it looks like the connections are shown as ESTABLISHED in netstat, even if I try opening connections to a process that does listen() but never accept(). – Phil Frost Dec 7 '16 at 20:33
  • Right, looking at the sources it seems those flags are for UNIX sockets. For TCP you could just count SYN_RECV though. There is no other queue beyond that. I suppose the kernel can be told somehow to log dropped packets because of too many half-open connections, but there have been some 10+ years since I looked at networking with Linux, so I have no idea how to do that. On a side note: you aren't waiting for accept() to do its job, you're waiting for ACKs to arrive from the connecting hosts to complete the connections. – Satō Katsura Dec 7 '16 at 20:47
1

Sysdig will provide some of this information at the end of each accept syscall, as the queuelen argument. It also shows the length of the queue as queuemax.

7598971 21:05:30.322229280 1 gunicorn (6451) < accept fd=13(<4t>127.0.0.1:45882->127.0.0.1:8003) tuple=127.0.0.1:45882->127.0.0.1:8003 queuepct=0 queuelen=0 queuemax=10

As far as I'm aware, it provides no mechanism to know exactly when or how many times the queue has overflowed. And it would be cumbersome to integrate this with periodic monitoring by collectd or similar.

0

What you are looking for is the entry in output of sysctl -a command as such:::

net.ipv4.tcp_max_sync_backlog = 4096

In the above example case, the backlog of SYN state connections is maximum 4096. You can increase that based on how much RAM is in your server. I consider 32K backlog to be a good start for tuning of heavily loaded web servers.

Also make sure the following is NOT set to One (1)::

net.ipv4.tcp_abort_on_overflow = 0

Otherwise it will definitely drop packets if there is a backlog overflow.

You can easily check via

"sysctl -a | egrep backlog"

"sysctl -a | egrep overflow"

Additionally, you can find "dropped" label under the

"ifconfig -a "

command's output. That shows how many packets were dropped for each interface along with other data and errors etc.

For Logging dropped packets there is a paywall article on RHEL 7::

https://access.redhat.com/solutions/1191593

For further research you may read:

http://veithen.io/2014/01/01/how-tcp-backlog-works-in-linux.html

It states here as per Steven's Book Illustrated TCP/IP:

"The queue limit applies to the sum of […] the number of entries on the incomplete connection queue […] and […] the number of entries on the completed connection queue […]."

Hence also states that:

"The completed connection queue is almost always empty because when an entry is placed on this queue, the server’s call to accept returns, and the server takes the completed connection off the queue."

The accept queue may hence seem completely empty and you will have to tune your (possibly in this case) Web Apache server to accept faster the connections placed on the "total aggregate" queue.

  • While there seems to be some useful information here, I’m not sure it answers the question.  If I ask, “What’s the most number of people that have ever been in this auditorium at one time?”, and you point to a sign on the wall that gives the maximum capacity, you haven’t answered the question. – Scott Jun 8 at 15:15
  • Indeed I'm looking for the current length of the queue, not the maximum length of the queue. – Phil Frost Jun 8 at 18:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.