1

I am working on a machine (ubuntu 18.04) that generates logs at times when it's not connected to a network. I have configured rsyslog to forward some of these logs to an aggregator service (fluentd) when the machine gets plugged into a network. The configuration looks like this:

module(load="imfile")
ruleset(name="sendtofluentd") {
    action(
        type="omfwd"
        target="example.com"                        
        port="6666"
        protocol="tcp"
        queue.type="LinkedList"
        queue.size="100000"                     
        queue.filename="skidder-queue"      
        queue.saveonshutdown="on"               
        queue.maxdiskspace="5g"             
        action.resumeRetryCount="-1"            
        action.resumeInterval="30"              
        action.resumeIntervalMax="99999999999"  
    )
}
input(type="imfile" tag="camera.falcon" file="/var/log/myapp/*.log" ruleset="sendtofluentd")
input(type="imfile" tag="camera.syslog" file="/var/log/syslog" ruleset="sendtofluentd")

Things are working ok, but I have a couple of questions.

  1. I see that rsyslog is spooling queue files at /var/spool/rsyslog/myapp.xxxxxxx. I see this file growing to hundreds of gigabytes. I'm concerned that these queue files will just keep growing and I might run out of disk space. It seems like a lot of these files have made it to the "sendtofluentd" action. Once they are there, I don't need these queue files anymore. Will I just need to occasionally rm these files? Will that cause any problems with rsyslog? Should I keep /var/spool/rsyslog/myapp.qi?
  2. If there are existing files at /var/log/myapp/*.log that were there before rsyslog started, is it going to attempt to send those logs to the queue, or is it more like a tail, where it only watches files that are changing during the time rsyslog is running?

Thank you for reading.

  • For question 2, each time you restart rsyslogd it will read and send the whole of any existing files. You should add PersistStateInterval=999 to have it create a state file (updated every 999 lines read) to remember how far it has read in case of a restart (or =0 if only clean shutdowns). see imfile. – meuh Jan 13 at 21:44
  • There are some queue examples in the redhat admin guide that might provide an answer, but in principle, if rsyslog creates a file for a queue I would expect it to do the housekeeping needed to remove it when appropriate, but I have no experience in this case. – meuh Jan 13 at 21:53
  • Thank you very much for your advice, @meuh. I appreciate you taking the time to read my questions and respond. It was helpful. – Lee Gaines Jan 15 at 0:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.