How can I limit the size of a log file written with
>> to 200MB?
$ run_program >> myprogram.log
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. It only takes a minute to sign up.Sign up to join this community
If your application (ie.
run_program) does not support limiting the size of the log file, then you can check the file size periodically in a loop with an external application or script.
You can also use
logrotate(8) to rotate your logs, it has
size parameter which you can use for your purpose:
With this, the log file is rotated when the specified size is reached. Size may be specified in bytes (default), kilobytes (sizek), or megabytes (sizem).
If your program doesn't need to write any OTHER files that would be larger than this limit, you can inform the kernel of this limit using
ulimit. Before you run your command, run this to setup a 200MB file size limit for all process run in your current shell session:
ulimit -f $((200*1024))
This will protect your system but it might be jaring for the program writing the file. As eyazici suggests, consider setting up
logrotate to prune log files once they reach a certain size or age. You can discard old data or archive it for a period of time in a series of compressed files.
You may create a new filesystem image, mount it using loop device and put the log file on that filesystem:
dd if=/dev/zero of=./200mb.img bs=1024 count=200000 # create new empty 200MB file mkfs.ext2 200mb.img # or ext3, or whatever fits your needs mkdir logs sudo mount -t ext2 -o loop 200mb.img logs # only root can do '-o loop' by default run_program >>logs/myprogram.log
You may also use
tmpfs instead of a file, if you have enough memory.
apache2-utils is present utility called
rotatelogs, it may be helpful for you.
rotatelogs [ -l ] [ -L linkname ] [ -p program ] [ -f ] [ -t ] [ -v ] [ -e ] [ -c ] [ -n number-of-files ] logfile rotationtime|filesize(B|K|M|G) [ offset ]
your_program | rotatelogs -n 5 /var/log/logfile 1M
Full manual you may read on this link.
I'm certain the original poster has found a solution. Here's another one for others that may read this thread...
Curtail limits the size of a program's output and preserves the last 200MB of output with the following command:
$ run_program | curtail -s 200M myprogram.log
NOTE: I'm the maintainer of the above repo. Just sharing the solution...
The following script should do the job.
LOG_SIZE=500000 NUM_SEGM=2 while getopts "s:n:" opt; do case "$opt" in s) LOG_SIZE=$OPTARG ;; n) NUM_SEGM=$OPTARG ;; esac done shift $((OPTIND-1)) if [ $# == 0 -o -z "$1" ]; then echo "missing output file argument" exit 1 fi OUT_FILE=$1 shift NUM=1 while :; do dd bs=10 count=$(($LOG_SIZE/10)) >> $OUT_FILE 2>/dev/null SZ=`stat -c%s $OUT_FILE` if [ $SZ -eq 0 ]; then rm $OUT_FILE break fi echo -e "\nLog portion finished" >> $OUT_FILE mv $OUT_FILE $OUT_FILE.n$NUM NUM=$(($NUM + 1)) [ $NUM -gt $NUM_SEGM ] && NUM=1 done
It has a couple of obvious short-cuts, but overall it does what you asked for. It will split the log into a chunks of a limited size, and the amount of chunks is limited too. All can be specified via the command-line arguments. Log file is also specified via the command line.
Note a small gotcha if you use it with the daemon that forks into background. Using a pipe will prevent the daemon from going to background. In this case there is a (likely bash-specific) syntax to avoid the problem:
my_daemon | ( logger.sh /var/log/my_log.log <&0 & )
<&0, while seemingly redundant, it won't work without this.