I get a few rcv pruned packets on my load balancers a day and I would like to know what the cause is:

enter image description here

However, based on my tcp_mem vs tcp_mem usage I don't understand why some packets are being pruned:

cat /proc/sys/net/ipv4/tcp_mem
6178080 8237440 12356160

enter image description here

  • What is the value of /proc/sys/net/ipv4/tcp_rmem ? You might consider increasing the initial buffer size and see if that prevents buffer overruns (i.e pruning). – Bratchley Apr 29 '14 at 14:48
  • Is this really causing an issue for you or were you just trying to make sense of the graph? – Bratchley Apr 29 '14 at 14:51
  • @JoelDavis: It is our load balancer, so it could mean a few people are getting a slower page load when loading these sites a day, so super critical! (Okay, it probably isn't a big deal, but "the more you know" – Kyle Brandt Apr 29 '14 at 14:52
  • @JoelDavis: And tcp_rmem is 183936 245248 367872, so I should still be okay unless there are some spikes between my polling of tcp_mem (which is every 15 seconds) – Kyle Brandt Apr 29 '14 at 14:53
  • You may try playing around with the different TCP congestion algorithms linux has available. It might just be an issue with congestion control. – Bratchley Apr 29 '14 at 14:58

The TCP RcvPruned MIB is incremented just before the kernel leaves the tcp_prune_queue() function, resulting in dropping of packets.

Prior to this, the kernel will have tries to collapse data in socket buffers (squeeze data together and discard overheads). Data will first be pruned from the out-of-order queue, then from the regular in-order queue.

This means your sockets are facing memory pressure, likely your socket buffers are not big enough.

You said above:

net.ipv4.tcp_rmem = 183936 245248 367872

The maximum is nowhere near big enough. Increase it to 10x that value to give the kernel room to increase socket memory as it wants to. Increase net.core.rmem_max as well, as that is used in TCP window accounting.

I'm not sure how you've calculated your buffer sizes, but if you're load balancing lots of small-packet traffic (like HTTP requests) then most of your buffer usage will be overhead.

Depending on which kernel you're running, there is different accounting for this. The "old" method makes some inaccurate assumptions but turns out to be quite forgiving, the "new" method accounts accurately and requires buffers to be sized a bit larger than previous.

The patch set which changes this focuses around the calculation of skb->truesize and IIRC was applied sometime between kernel 3.0 and 3.10.

You also will want to check the value of net.ipv4.tcp_mem and make sure you're not restricting the amount of total memory TCP is allowed to use. This tunable is measured in pages, not bytes. Personally I just make all values so big they'll never be hit. Buy more memory as required.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.