2

I have a couple Oracle Linux 6 systems acting as rsyslogd remote receivers. I'm pretty much certain the TCP stack is overloaded, and I've been trying to tune it. But, I'm at the end of my Google searches, and I don't know what to try next. I've been through these sites already

along with a bunch of uselessness. Here's the tuning I've tried so far:

net.core.wmem_max = 12582912
net.core.rmem_max = 12582912
net.core.netdev_max_backlog = 5000
net.ipv4.tcp_rmem = 10240 87380 12582912
net.ipv4.tcp_wmem = 10240 87380 12582912
net.ipv4.tcp_max_syn_backlog = 2048

And, I'll still get unanswered packets coming in. (The source is a test application just sending zero-length packets.)

2017-12-12 15:25:41.053937 IP SENDER.34394 > RECEIVER.514: Flags [S], seq 1418245921, win 18352, options [mss 9176,sackOK,TS val 1117377413 ecr 0,nop,wscale 7], length 0
2017-12-12 15:25:42.053458 IP SENDER.34394 > RECEIVER.514: Flags [S], seq 1418245921, win 18352, options [mss 9176,sackOK,TS val 1117378413 ecr 0,nop,wscale 7], length 0
2017-12-12 15:25:44.053578 IP SENDER.34394 > RECEIVER.514: Flags [S], seq 1418245921, win 18352, options [mss 9176,sackOK,TS val 1117380413 ecr 0,nop,wscale 7], length 0
2017-12-12 15:25:48.053582 IP SENDER.34394 > RECEIVER.514: Flags [S], seq 1418245921, win 18352, options [mss 9176,sackOK,TS val 1117384413 ecr 0,nop,wscale 7], length 0
2017-12-12 15:25:56.053803 IP SENDER.34394 > RECEIVER.514: Flags [S], seq 1418245921, win 18352, options [mss 9176,sackOK,TS val 1117392413 ecr 0,nop,wscale 7], length 0
2017-12-12 15:26:12.055312 IP SENDER.34394 > RECEIVER.514: Flags [S], seq 4104732283, win 18352, options [mss 9176,sackOK,TS val 1117408413 ecr 0,nop,wscale 7], length 0

And, if my receiver does answer the incoming requests, it seems to ignore the test FIN flags for several tries before it RSTs the connection:

2017-12-12 15:25:55.632684 IP SENDER.56008 > RECEIVER.514: Flags [S], seq 3551538126, win 18352, options [mss 9176,sackOK,TS val 1117391991 ecr 0,nop,wscale 7], length 0
2017-12-12 15:25:56.632380 IP SENDER.56008 > RECEIVER.514: Flags [S], seq 3551538126, win 18352, options [mss 9176,sackOK,TS val 1117392991 ecr 0,nop,wscale 7], length 0
2017-12-12 15:25:58.631714 IP SENDER.56008 > RECEIVER.514: Flags [S], seq 3551538126, win 18352, options [mss 9176,sackOK,TS val 1117394991 ecr 0,nop,wscale 7], length 0
2017-12-12 15:26:02.632298 IP SENDER.56008 > RECEIVER.514: Flags [S], seq 3551538126, win 18352, options [mss 9176,sackOK,TS val 1117398991 ecr 0,nop,wscale 7], length 0
2017-12-12 15:26:10.632477 IP SENDER.56008 > RECEIVER.514: Flags [S], seq 3551538126, win 18352, options [mss 9176,sackOK,TS val 1117406991 ecr 0,nop,wscale 7], length 0
2017-12-12 15:26:10.632517 IP RECEIVER.514 > SENDER.56008: Flags [S.], seq 1010914189, ack 3551538127, win 14480, options [mss 1460,sackOK,TS val 1002368883 ecr 1117406991,nop,wscale 8], length 0
2017-12-12 15:26:10.632983 IP SENDER.56008 > RECEIVER.514: Flags [.], ack 1, win 144, options [nop,nop,TS val 1117406991 ecr 1002368883], length 0
2017-12-12 15:26:10.664428 IP SENDER.56008 > RECEIVER.514: Flags [F.], seq 1, ack 1, win 144, options [nop,nop,TS val 1117407023 ecr 1002368883], length 0
2017-12-12 15:26:10.865080 IP SENDER.56008 > RECEIVER.514: Flags [F.], seq 1, ack 1, win 144, options [nop,nop,TS val 1117407224 ecr 1002368883], length 0
2017-12-12 15:26:11.267436 IP SENDER.56008 > RECEIVER.514: Flags [F.], seq 1, ack 1, win 144, options [nop,nop,TS val 1117407626 ecr 1002368883], length 0
2017-12-12 15:26:11.772951 IP RECEIVER.514 > SENDER.56008: Flags [S.], seq 1010914189, ack 3551538127, win 14480, options [mss 1460,sackOK,TS val 1002370024 ecr 1117407626,nop,wscale 8], length 0
2017-12-12 15:26:11.773690 IP SENDER.56008 > RECEIVER.514: Flags [.], ack 1, win 144, options [nop,nop,TS val 1117408132 ecr 1002368883], length 0
2017-12-12 15:26:12.071025 IP SENDER.56008 > RECEIVER.514: Flags [F.], seq 1, ack 1, win 144, options [nop,nop,TS val 1117408430 ecr 1002368883], length 0
2017-12-12 15:26:13.679547 IP SENDER.56008 > RECEIVER.514: Flags [F.], seq 1, ack 1, win 144, options [nop,nop,TS val 1117410038 ecr 1002368883], length 0
2017-12-12 15:26:13.772915 IP RECEIVER.514 > SENDER.56008: Flags [S.], seq 1010914189, ack 3551538127, win 14480, options [mss 1460,sackOK,TS val 1002372024 ecr 1117410038,nop,wscale 8], length 0
2017-12-12 15:26:13.773784 IP SENDER.56008 > RECEIVER.514: Flags [.], ack 1, win 144, options [nop,nop,TS val 1117410132 ecr 1002368883], length 0
2017-12-12 15:26:16.895194 IP SENDER.56008 > RECEIVER.514: Flags [F.], seq 1, ack 1, win 144, options [nop,nop,TS val 1117413254 ecr 1002368883], length 0
2017-12-12 15:26:23.327789 IP SENDER.56008 > RECEIVER.514: Flags [F.], seq 1, ack 1, win 144, options [nop,nop,TS val 1117419686 ecr 1002368883], length 0
2017-12-12 15:26:23.327833 IP RECEIVER.514 > SENDER.56008: Flags [R], seq 1010914190, win 0, length 0

So, I think I'm still backlogged, in that my TCP stack isn't getting to all the incoming packets as fast as it should. But, I don't want to start messing with further settings without some better documentation references and a better understand of how the TCP stack settings interact with each other.

For bonus points, does anyone have a utility better than netstat -i to track the TCP stack in real time?

UPDATES:

  • These hosts are not load-balanced, but I do have keepalived running a Virtual IP address, so I get a bit of failover if the primary system dies.
  • These hosts are OL6 VMs on top of VMware.
  • Unfortunately, I cannot help with the primary question itself, but as far as the secondary question, a good network monitoring and statistics gathering tool worth checking out is iptraf-ng. – Travis Clarke Dec 13 '17 at 5:00
  • Tuning is a black art. Also have a look at tc (traffic control) if priorizing packets can help. – dirkt Dec 13 '17 at 7:41
  • First tuning is hardware, and I think rsyslogd is the slower part (or you have crap network hardware). – Giacomo Catenazzi Dec 13 '17 at 11:30
  • Is your box behind a load balancer? Check this vincent.bernat.im/en/blog/2014-tcp-time-wait-state-linux.html. How many new connection requests per second is your server receiving? How long a single connection takes to be served? Are you sure that your hardware and application can handle the current load? Take a look at this page if you didn't please kegel.com/c10k.html – Luciano Afranllie Dec 13 '17 at 17:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.