I have a system that is acting simultaneously as a test and demo platform. I'm recording a massive amount of data output by my systems that I wouldn't be recording when this is in deployment and because of that my log files are filling up very rapidly and I eventually run out of partition space on my harddrive.

I tried logrotate, but it seems to only work if my program isn't running. When the programs are down it properly truncates the file, records a compressed version, and discards the rest. However, when my programs that populate the log file are up and running it will properly create the compressed file, but it won't truncate the active file. I'm trying to figure out how to make it truncate.

I'm writing to the log files with a simple redirect. ProgramA>logA, programB>logB etc. My progam is outputting every time I receive an input vector, so it's outputting very rapidly. My assumption is that the logrotate is failing to truncate because of the constant writes to the field, but can anyone confirm rather that is the reason?

Where does logrotate keep it's error file anyways?

Also: I want to modify the behavior of logrotate. If it activates and sees a large, say 100 MB, log file I would like it to create a copy of only the last 1 MB of the file (discarding the older 99 MB of content of the file) before truncating the original log file. I need to keep recent data, but I don't care much about anything that is very old. Can anyone tell me how to do this?

  • Have you considered cronolog? Pipe to that instead your current redirect, and it seems like that'd solve a lot of your problems. Also, if you really only care about the last 1MB, it sounds like you need a ring buffer logger of some sort.
    – derobert
    Commented Aug 28, 2012 at 22:34
  • I'll look into cronolog right now, never heard of it before. Since the current 'logs' are all debug output that will be disabled in final release I was trying not to waste too much time writing a complicated system for handling the logging. If you know a linux program that will do the ring-buffering for me without my implementing I would definately look at it; but I don't want to write the logic for one.
    – dsollen
    Commented Aug 29, 2012 at 0:24
  • You could write a close-enough to ring buffer program in a few lines of Perl. In fact, I think I shall do so as an answer...
    – derobert
    Commented Aug 29, 2012 at 1:13
  • Also, I just found serverfault.com/questions/133320/… ... which has some more approaches. Including a kernel module (wtf).
    – derobert
    Commented Aug 29, 2012 at 17:24
  • NOTE for the unwary: Most operating systems will let you delete and rename a file (such as a log file) even while another program still has it open and is still writing to it. So it may still be possible that an opened log file will continue to grow in size even when you try to rename/rotate them. And trying to truncate the log will likely cause the program writing to the log to do strange things because it is not expecting the log file to suddenly become empty. Caveat emptor.
    – C. M.
    Commented Apr 22, 2021 at 12:46

3 Answers 3


I know this is a little bit old but I had the same problem, and the solution was to use the append redirection to the output file rather than clobbering it. This way your application/script wont lock the file exclusively .

Instead of:

ProgramA > logA


ProgramA >> logA 


  • idk, just trial and error. :\ Somehow it makes sense for me.
    – vegatripy
    Commented Apr 11, 2016 at 16:55
  • This one worked for me
    – sinsuren
    Commented Aug 31, 2017 at 15:08

From man logrotate:

          Truncate the original log file to zero size in place after creat‐
          ing a copy, instead of moving the old  log  file  and  optionally
          creating  a  new one.  It can be used when some program cannot be
          told to  close  its  logfile  and  thus  might  continue  writing
          (appending) to the previous log file forever.  Note that there is
          a very small time slice between copying the file  and  truncating
          it,  so  some  logging  data  might be lost.  When this option is
          used, the create option will have no effect, as the old log  file
          stays in place.

That's probably the option you need to put in your configuration file.

About your other question: use the postrotate directive to run a command to truncate the file after it has been rotated. In that case, you should compress it manually at this stage, after truncating. (not really, this last part, because logrotate doesn't compress the first archive.)

EDIT: A little more about keeping the last 1 MB. You can configure it to rotate when the log file has more than, say, 200 K, and to only keep 5 backups. That way you'll always keep roughly the last 1 MB. If the first backup is too big, you can just wait until it "falls off" after 5 backups are done.

  • I suppose I should have clearified. I am using copytruncate command. However, it doesn't do the truncat as requested unless my programs are dead.
    – dsollen
    Commented Aug 29, 2012 at 0:22
  • @dsollen Just redirect the output of your programs with >> instead of >. copytruncate will work then.
    – angus
    Commented Aug 29, 2012 at 8:42

As promised, a "close enough" to ring buffer program, in a few lines of Perl.

To use it, pass it the size of the log files you want, in bytes (give or take one line), and two or more log files. It'll switch between the log files every X bytes. I'm sure there is already a program out there to do this, but I couldn't find it with a quick search. E.g.,

$ your-app | log-splitter 1048576 /var/log/your-app/log1 /var/log/your-app/log2

assuming, of course, that the file log-splitter is executable, in $PATH, and contains:

use warnings qw(all);
use strict;
use autodie;
use Carp;
use Fcntl qw(SEEK_SET);
use IO::Handle;

# yes, we truncate all logs at start. Sorry.
@ARGV >= 2 or croak "Usage: $0 log-size log-file...";
my ($max_size, @lognames) = @ARGV;
my @logs = map {
    open my $fh, '+>', $_;
} @lognames;

my $current_size = $max_size;
my $current_log  = $#logs;
while (defined(my $line = <STDIN>)) {
    if ($current_size >= $max_size) {
        $current_log  = ($current_log + 1) % @logs;
        $current_size = 0;
        seek($logs[$current_log], 0, SEEK_SET);
        truncate($logs[$current_log], 0);

    $current_size += length($line);
    print {$logs[$current_log]} $line;

close($_) foreach @logs;

To clarify any licensing questions on that code:

To the extent possible under law, Anthony DeRobertis has waived all copyright and related or neighboring rights to the above program (log-splitter). This work is published from: United States.

  • Well, it's true that replacing > file with >> file is lazier than replacing it with | log-splitter 1234 file1 file2, but I would have counted that as an advantage :D
    – angus
    Commented Aug 29, 2012 at 19:15

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .