How to limit log file size using >>

Question

How can I limit the size of a log file written with >> to 200MB?

$ run_program >> myprogram.log

Answer 1

If your application (ie. run_program) does not support limiting the size of the log file, then you can check the file size periodically in a loop with an external application or script.

You can also use logrotate(8) to rotate your logs, it has size parameter which you can use for your purpose:

With this, the log file is rotated when the specified size is reached. Size may be specified in bytes (default), kilobytes (sizek), or megabytes (sizem).

Answer 2

If your program doesn't need to write any OTHER files that would be larger than this limit, you can inform the kernel of this limit using ulimit. Before you run your command, run this to setup a 200MB file size limit for all process run in your current shell session:

ulimit -f $((200*1024))

This will protect your system but it might be jaring for the program writing the file. As eyazici suggests, consider setting up logrotate to prune log files once they reach a certain size or age. You can discard old data or archive it for a period of time in a series of compressed files.

Answer 3

You may create a new filesystem image, mount it using loop device and put the log file on that filesystem:

dd if=/dev/zero of=./200mb.img bs=1024 count=200000 # create new empty 200MB file
mkfs.ext2 200mb.img # or ext3, or whatever fits your needs
mkdir logs
sudo mount -t ext2 -o loop 200mb.img logs # only root can do '-o loop' by default
run_program >>logs/myprogram.log

You may also use tmpfs instead of a file, if you have enough memory.

Answer 4

You can truncate the output with head:

size=$((200*1024*1024-$(stat -c %s myprogram.log)))
run_program | head -c ${size} >> myprogram.log

Answer 5

In package apache2-utils is present utility called rotatelogs, it may be helpful for you.

Synopsis:

rotatelogs [ -l ] [ -L linkname ] [ -p program ] [ -f ] [ -t ] [ -v ] [ -e ] [ -c ] [ -n number-of-files ] logfile rotationtime|filesize(B|K|M|G) [ offset ]

Example:

your_program | rotatelogs -n 5 /var/log/logfile 1M

Full manual you may read on this link.

Answer 6

I'm certain the original poster has found a solution. Here's another one for others that may read this thread...

Curtail limits the size of a program's output and preserves the last 200MB of output with the following command:

$ run_program | curtail -s 200M myprogram.log

References

• https://github.com/Comcast/Infinite-File-Curtailer

NOTE: I'm the maintainer of the above repo. Just sharing the solution...

Answer 7

Since it is text, I would write a script in your favorite language and pipe it to that. Have it handle the file I/O (or keep it all in memory and then dump it on SIGHUP or similar). For that, instead of 200MB I would think of a 'reasonable' number of lines to keep track of.

Answer 8

The following script should do the job.

LOG_SIZE=500000
NUM_SEGM=2
while getopts "s:n:" opt; do
  case "$opt" in
    s)
      LOG_SIZE=$OPTARG
      ;;
    n)
      NUM_SEGM=$OPTARG
      ;;
  esac
done
shift $((OPTIND-1))
if [ $# == 0 -o -z "$1" ]; then
    echo "missing output file argument"
    exit 1
fi
OUT_FILE=$1
shift
NUM=1
while :; do
    dd bs=10 count=$(($LOG_SIZE/10)) >> $OUT_FILE 2>/dev/null
    SZ=`stat -c%s $OUT_FILE`
    if [ $SZ -eq 0 ]; then
        rm $OUT_FILE
        break
    fi
    echo -e "\nLog portion finished" >> $OUT_FILE
    mv $OUT_FILE $OUT_FILE.n$NUM
    NUM=$(($NUM + 1))
    [ $NUM -gt $NUM_SEGM ] && NUM=1
done

It has a couple of obvious short-cuts, but overall it does what you asked for. It will split the log into a chunks of a limited size, and the amount of chunks is limited too. All can be specified via the command-line arguments. Log file is also specified via the command line.

Note a small gotcha if you use it with the daemon that forks into background. Using a pipe will prevent the daemon from going to background. In this case there is a (likely bash-specific) syntax to avoid the problem:

my_daemon | ( logger.sh /var/log/my_log.log <&0 & )

Note the <&0, while seemingly redundant, it won't work without this.