1

We are monitoring a few Linux servers (mostly CentOS 8) that aren't administrated by us, but we're responsible for providing the VMs. These servers have rsyslog installed and are sending messages to our central syslog server. For a while now the disk space fills up regularly, and it takes a while for the responsible person to free up some space. However, we noticed that rsyslog is creating a massive amount of messages while the disk is full. We are talking about ~100 messages per second and around 20k messages for every 10 minute interval just for these messages and all from one machine (they all repeat, there are no other messages in between these):

Rsyslog spamming our syslog server

I can't imagine that this is meant to be like this. I'm not very familiar with rsyslog, but I couldn't find anything of interest in the configuration file (/etc/rsyslog.conf, /etc/rsyslog.d was empty). I would like to reduce the amount of messages to ~1 per minute or every few minutes, so the messages won't get buried either. Apparently I can rate limit the messages, but I'm worried that "legitimate" messages could get lost this way. Is there some way to prevent this and rate limit the messages in a previous step?

Thank you.

4
  • It probably is intended to be like this -- there is a log message for every failed write. There is an obvious common cause, but the detail is intended to identify which files may be output by runaway processes, and which other processes may have tainted or lost output through no fault of their own. (All processes should check for disk full on every write, but few do.) Production systems should monitor %age of disk space (and inodes) available), and provide early warning. Extra points on alerts for sudden growth of usage. Feb 15, 2021 at 18:25
  • @Paul_Pedant >It probably is intended to be like this -- there is a log message for every failed write That is the thing, this machine usually only writes around 10-20 logs in one hour, so 100 messages per second seem very excessive.
    – lauraaaaa
    Feb 16, 2021 at 7:09
  • it simply logs one message per serious event. If you fill up /var/log/... (probably / itself) that is a system-wide situation that fouls up every process that is writing a file, including syslog itself. You might look at what happened just before those messages started -- e.g. a memory fault or RAID failure if syslog itself is huge, or the first program that fills /. Throttling syslog instead of fixing the root cause is just shooting the messenger for bringing bad news -- somewhat irresponsible. Use du to find huge files, and see who the owner is. Feb 16, 2021 at 14:59
  • 1
    I know this is just fighting the symptoms, but, as I mentioned, we only provide the infrastructure. Other people are administrating the machines and we don't have in-depth knowledge about the services running on them. Of course, this situation isn't optimal, but I can't change anything about it. We notify the service owners when things like this happen, but apparently they can't be bothered to fix the reasons (or are not capable to).
    – lauraaaaa
    Feb 16, 2021 at 15:12

1 Answer 1

0

From the global object description you can provide a separate limit for internally generated error messages with, eg:

global(internalmsg.ratelimit.interval=5
       internalmsg.ratelimit.burst=10)

which would only show at most 10 such messages per 5 second interval.

2
  • Thank you, this is exactly what I needed.
    – lauraaaaa
    Feb 16, 2021 at 15:09
  • 1
    On rsyslogd version 8.2302.0, I had the error invalid character '5' in object definition. Fixed it by enclosing the integer values between double quotes, e.g. internalmsg.ratelimit.burst="10".
    – Gus
    Oct 3, 2023 at 12:34

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .