3

I'm trying to set up a Heartbeat client on a linux machine (CentOS Linux release 7.3.1611) which is sending ICMP echo requests to roughly 1300 hosts. However, in future this number will be higher. The messages generated by heartbeat are being shipped to a logstash instance on another server.

These requests go out every 3 minutes, all at once. After the first round of requests, a large number of these messages contain an error:

"write ip4 0.0.0.0->x.x.x.x: sendto: no buffer space available"

I've tried multiple ways of mitigating this issue, including staggering the requests so there are less being sent out in 1 second. This caused issues with data synchronisation at the end of this pipeline (as well as not completely resolving the issue) so it's not really a feasible option for me.

Researching online led me to increasing the memory allocated to TCP/IP connections by the system, however these are ICMP requests which as far as I'm aware should be separate.

I'm also unable to bounce the network interface as the machine is remote and I don't have a way of getting back in to bring it back up.

I also tested out increasing (and setting to 0) the icmp_ratelimit variable in /proc/sys/net/ipv4 but this hasn't helped either.

My question is pretty general, what could be causing this issue? Is there some sort of variable I need to tweak on the system that will allow these requests to all be sent out at once? I can't really deduce what buffer the error message is referencing.

Any help would be greatly appreciated...

P.S. if further clarification is needed I'd be happy to provide it.

EDIT


Still unsure if there is a fault elsewhere however increasing the size of the socket buffer determined by the wmem_max, wmem_default, rem_max, rem_default variables in /proc/sys/net/core has fixed the issue. It's likely that the total size of the data for all the ICMP echo requests was too large to fit under the previous maximum of 208kb and so a large number of them would be dropped. Doesn't really explain why the number of dropped requests would vary each time, maybe there is an underlying issue...

Now my only problem is that each time I reboot the system, these variables reset to 208kb each. How do I make these changes persistent?

  • Looks like a kernel issue. Does ifdown/ifup fix it? If yes, try a kernel upgrade if it is old. – peterh Aug 15 '17 at 8:58
  • Could be a NIC driver/offloading bug. What kind of NIC is this? Maybe try to turn various kinds of offloading on or off (using ethtool -k). Also, see if ip link set eth0 qlen 10000 improves matters. – András Korn Aug 15 '17 at 10:39
  • @peterh Not really able to try that as SSH is my only real way of connecting to this box, wouldn't be able to get back in to bring the interface back up. A luxury I didn't have unfortunately. – P.Ackland Aug 15 '17 at 12:34
  • Then it is likely not a kernel issue, or not a very serious one, because your ssh remains intact. – peterh Aug 15 '17 at 12:39
  • To persist changes see man 8 sysctl – meuh Aug 15 '17 at 14:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.