Logrotate not truncating open file

Question

I have a system that is acting simultaneously as a test and demo platform. I'm recording a massive amount of data output by my systems that I wouldn't be recording when this is in deployment and because of that my log files are filling up very rapidly and I eventually run out of partition space on my harddrive.

I tried logrotate, but it seems to only work if my program isn't running. When the programs are down it properly truncates the file, records a compressed version, and discards the rest. However, when my programs that populate the log file are up and running it will properly create the compressed file, but it won't truncate the active file. I'm trying to figure out how to make it truncate.

I'm writing to the log files with a simple redirect. ProgramA>logA, programB>logB etc. My progam is outputting every time I receive an input vector, so it's outputting very rapidly. My assumption is that the logrotate is failing to truncate because of the constant writes to the field, but can anyone confirm rather that is the reason?

Where does logrotate keep it's error file anyways?

Also: I want to modify the behavior of logrotate. If it activates and sees a large, say 100 MB, log file I would like it to create a copy of only the last 1 MB of the file (discarding the older 99 MB of content of the file) before truncating the original log file. I need to keep recent data, but I don't care much about anything that is very old. Can anyone tell me how to do this?

Answer 1

I know this is a little bit old but I had the same problem, and the solution was to use the append redirection to the output file rather than clobbering it. This way your application/script wont lock the file exclusively .

Instead of:

ProgramA > logA

Use:

ProgramA >> logA 

Regards.

Answer 2

From man logrotate:

   copytruncate
          Truncate the original log file to zero size in place after creat‐
          ing a copy, instead of moving the old  log  file  and  optionally
          creating  a  new one.  It can be used when some program cannot be
          told to  close  its  logfile  and  thus  might  continue  writing
          (appending) to the previous log file forever.  Note that there is
          a very small time slice between copying the file  and  truncating
          it,  so  some  logging  data  might be lost.  When this option is
          used, the create option will have no effect, as the old log  file
          stays in place.

That's probably the option you need to put in your configuration file.

About your other question: use the postrotate directive to run a command to truncate the file after it has been rotated. In that case, you should compress it manually at this stage, after truncating. (not really, this last part, because logrotate doesn't compress the first archive.)

EDIT: A little more about keeping the last 1 MB. You can configure it to rotate when the log file has more than, say, 200 K, and to only keep 5 backups. That way you'll always keep roughly the last 1 MB. If the first backup is too big, you can just wait until it "falls off" after 5 backups are done.

Answer 3

As promised, a "close enough" to ring buffer program, in a few lines of Perl.

To use it, pass it the size of the log files you want, in bytes (give or take one line), and two or more log files. It'll switch between the log files every X bytes. I'm sure there is already a program out there to do this, but I couldn't find it with a quick search. E.g.,

$ your-app | log-splitter 1048576 /var/log/your-app/log1 /var/log/your-app/log2

assuming, of course, that the file log-splitter is executable, in $PATH, and contains:

#!/usr/bin/perl
use warnings qw(all);
use strict;
use autodie;
use Carp;
use Fcntl qw(SEEK_SET);
use IO::Handle;

# yes, we truncate all logs at start. Sorry.
@ARGV >= 2 or croak "Usage: $0 log-size log-file...";
my ($max_size, @lognames) = @ARGV;
my @logs = map {
    open my $fh, '+>', $_;
    $fh->autoflush(1);
    $fh;
} @lognames;

my $current_size = $max_size;
my $current_log  = $#logs;
while (defined(my $line = <STDIN>)) {
    if ($current_size >= $max_size) {
        $current_log  = ($current_log + 1) % @logs;
        $current_size = 0;
        seek($logs[$current_log], 0, SEEK_SET);
        truncate($logs[$current_log], 0);
    }

    $current_size += length($line);
    print {$logs[$current_log]} $line;
}

close($_) foreach @logs;

To clarify any licensing questions on that code:

To the extent possible under law, Anthony DeRobertis has waived all copyright and related or neighboring rights to the above program (log-splitter). This work is published from: United States.