Log file rotation and compression interval

Question

We have our the logrotate config for a service as below:

{
    rotate 30
    create 644 root syslog
    missingok
    notifempty
    daily
    sharedscripts
    postrotate
        /bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true
    endscript }

Even though compress is not mentioned in the config, the log files are gzipped after each rotation. I believe it is because the compress line is uncommented in the /etc/logrotate.conf file, there by enabling it globally. The questions are:

1. Is there a time delay or interval between the log file is rotated, (from debug.log to debug.log-20190315) and when it is compressed (from debug.log-20190315 to debug.log-20190315.gz)?

2. If there is a delay, would mentioning compress in the specific log rotate config file of the service would compress that log file immediately after rotating it to debug.log-20190315 from debug.log ?

I do not see delaycompress mentioned in any of the logrotate config files.

(Background: Our Splunk Indexer seems to be indexing a debug.log-2019xxxx file from this service. We have blacklisted *.gz$ and debug.log$ from going to the Splunk, but it seems that the file debug.log-2019xxxx exists for a few seconds or minutes because of which it gets forwarded to Splunk since during that time it does not match the blacklisted regex - *.gz$ and debug.log$. I know that I can fix this by adding debug.log-[0-9]* to the blacklist, but would like to know what causes the existence of xxxxxx.debug.log-20190315)

Answer 1

The gzip compression step would take a finite amount of time, so it's possible that you're seeing the original file while the compression process is ongoing. If the size of the original file is large enough, it can exist for quite some time before the compression is completed. The time taken for the compression will vary according to the settings passed to gzip (speed & level of compression) as well as the compressibility of the file. gzip will remove the original file only after the compression process is complete.

As a small test for verification , I created a file of size 1 GiB from /dev/urandom and another file of size 1 GiB from /dev/zero and tested the time it took to compress them.

The file containing random data took about 2 minutes & 23 seconds:

[root@testvm1 ~]# time gzip testfile-random.txt

real    2m27.417s
user    2m22.172s
sys     0m2.839s

And the zero file took about 29 seconds:

[root@testvm1 ~]# time gzip testfile-zero.txt

real    0m28.930s
user    0m27.453s
sys     0m0.989s

While the compression was taking place, the original file was visible in both cases:

[root@testvm1 ~]# ls -lh testfile-random.txt*
-rw-r--r--. 1 root root 1.0G Mar 15 17:49 testfile-random.txt
-rw-------. 1 root root  75M Mar 15 17:59 testfile-random.txt.gz

[root@testvm1 ~]# ls -lh testfile-zero.txt*
-rw-r--r--. 1 root root 1.0G Mar 15 18:04 testfile-zero.txt
-rw-------. 1 root root 992K Mar 15 18:05 testfile-zero.txt.gz