We are monitoring a few Linux servers (mostly CentOS 8) that aren't administrated by us, but we're responsible for providing the VMs. These servers have rsyslog installed and are sending messages to our central syslog server. For a while now the disk space fills up regularly, and it takes a while for the responsible person to free up some space. However, we noticed that rsyslog is creating a massive amount of messages while the disk is full. We are talking about ~100 messages per second and around 20k messages for every 10 minute interval just for these messages and all from one machine (they all repeat, there are no other messages in between these):
I can't imagine that this is meant to be like this. I'm not very familiar with rsyslog, but I couldn't find anything of interest in the configuration file (/etc/rsyslog.conf, /etc/rsyslog.d was empty). I would like to reduce the amount of messages to ~1 per minute or every few minutes, so the messages won't get buried either. Apparently I can rate limit the messages, but I'm worried that "legitimate" messages could get lost this way. Is there some way to prevent this and rate limit the messages in a previous step?