How can I limit the size of a log file written with >>
to 200MB?
$ run_program >> myprogram.log
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. It only takes a minute to sign up.
Sign up to join this communityHow can I limit the size of a log file written with >>
to 200MB?
$ run_program >> myprogram.log
If your application (ie. run_program
) does not support limiting the size of the log file, then you can check the file size periodically in a loop with an external application or script.
You can also use logrotate(8)
to rotate your logs, it has size
parameter which you can use for your purpose:
With this, the log file is rotated when the specified size is reached. Size may be specified in bytes (default), kilobytes (sizek), or megabytes (sizem).
postscript
option to have the logrotate config send then SIGHUP
to the program.
Jul 23, 2011 at 21:48
If your program doesn't need to write any OTHER files that would be larger than this limit, you can inform the kernel of this limit using ulimit
. Before you run your command, run this to setup a 200MB file size limit for all process run in your current shell session:
ulimit -f $((200*1024))
This will protect your system but it might be jaring for the program writing the file. As eyazici suggests, consider setting up logrotate
to prune log files once they reach a certain size or age. You can discard old data or archive it for a period of time in a series of compressed files.
You may create a new filesystem image, mount it using loop device and put the log file on that filesystem:
dd if=/dev/zero of=./200mb.img bs=1024 count=200000 # create new empty 200MB file
mkfs.ext2 200mb.img # or ext3, or whatever fits your needs
mkdir logs
sudo mount -t ext2 -o loop 200mb.img logs # only root can do '-o loop' by default
run_program >>logs/myprogram.log
You may also use tmpfs
instead of a file, if you have enough memory.
You can truncate the output with head
:
size=$((200*1024*1024-$(stat -c %s myprogram.log)))
run_program | head -c ${size} >> myprogram.log
SIGPIPE
) once it has reached the size limit, rather than discarding the data.
Jul 22, 2011 at 11:38
dd
magic but yeah @Random832 is right, you'll get a SIGPIPE
as head
/dd
/whatever
drops it.
Jul 23, 2011 at 13:07
{ head -c "$size" >> log; cat > /dev/null; }
.
Jan 22, 2013 at 11:10
In package apache2-utils
is present utility called rotatelogs
, it may be helpful for you.
Synopsis:
rotatelogs [ -l ] [ -L linkname ] [ -p program ] [ -f ] [ -t ] [ -v ] [ -e ] [ -c ] [ -n number-of-files ] logfile rotationtime|filesize(B|K|M|G) [ offset ]
Example:
your_program | rotatelogs -n 5 /var/log/logfile 1M
Full manual you may read on this link.
I'm certain the original poster has found a solution. Here's another one for others that may read this thread...
Curtail limits the size of a program's output and preserves the last 200MB of output with the following command:
$ run_program | curtail -s 200M myprogram.log
NOTE: I'm the maintainer of the above repo. Just sharing the solution...
Since it is text, I would write a script in your favorite language and pipe it to that. Have it handle the file I/O (or keep it all in memory and then dump it on SIGHUP
or similar). For that, instead of 200MB I would think of a 'reasonable' number of lines to keep track of.
syslog
and logrotate
.
The following script should do the job.
LOG_SIZE=500000
NUM_SEGM=2
while getopts "s:n:" opt; do
case "$opt" in
s)
LOG_SIZE=$OPTARG
;;
n)
NUM_SEGM=$OPTARG
;;
esac
done
shift $((OPTIND-1))
if [ $# == 0 -o -z "$1" ]; then
echo "missing output file argument"
exit 1
fi
OUT_FILE=$1
shift
NUM=1
while :; do
dd bs=10 count=$(($LOG_SIZE/10)) >> $OUT_FILE 2>/dev/null
SZ=`stat -c%s $OUT_FILE`
if [ $SZ -eq 0 ]; then
rm $OUT_FILE
break
fi
echo -e "\nLog portion finished" >> $OUT_FILE
mv $OUT_FILE $OUT_FILE.n$NUM
NUM=$(($NUM + 1))
[ $NUM -gt $NUM_SEGM ] && NUM=1
done
It has a couple of obvious short-cuts, but overall it does what you asked for. It will split the log into a chunks of a limited size, and the amount of chunks is limited too. All can be specified via the command-line arguments. Log file is also specified via the command line.
Note a small gotcha if you use it with the daemon that forks into background. Using a pipe will prevent the daemon from going to background. In this case there is a (likely bash-specific) syntax to avoid the problem:
my_daemon | ( logger.sh /var/log/my_log.log <&0 & )
Note the <&0
, while seemingly redundant, it won't work without this.