Tell me more ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.
up vote 8 down vote favorite
1

How can I limit the size of a log file written with >> to 200MB?

$ run_program >> myprogram.log
share|improve this question
Do you want the program killed after 200MB? Or do you want the last 200MB with everything older in the bit bucket? – Aaron D. Marasco Jul 23 '11 at 13:10
nope, the process can't stop until manually killed – david Jul 25 '11 at 2:35
decided to go for logrotate, thanks everyone for the valuable inputs – david Jul 25 '11 at 3:18

5 Answers

up vote 6 down vote accepted

If your application (ie. run_program) does not support limiting the size of the log file, then you can check the file size periodically in a loop with an external application or script.

You can also use logrotate(8) to rotate your logs, it has size parameter which you can use for your purpose:

With this, the log file is rotated when the specified size is reached. Size may be specified in bytes (default), kilobytes (sizek), or megabytes (sizem).

share|improve this answer
1  
+1 Logfiles with repeated information can often get compressed in orders of magnitude. – user unknown Jul 22 '11 at 15:28
Does logrotate truncate files, or simply copy or move them? Because this will only work if the file associated with the fd is truncated. If it is mv'd, the file will continue to grow, if it is unlink'd, it will just be held open and continue to grow until the process exits... IIRC logrotate copies, unlinks, and creates a new file, which is why one often has to send SIGHUP to dæmons when their logs have been rotated. – Michael Trausch Jul 22 '11 at 16:26
Note that even if it's truncated, there's a problem if the file was not opened in append mode (log files should always be opened in append mode, but in my experience are often not) – Random832 Jul 22 '11 at 20:17
Logrotate can rotate once the file has reached a certain size, daily, weekly, etc. It can also compress, and you can use the postscript option to have the logrotate config send then SIGHUP to the program. – laebshade Jul 23 '11 at 21:48
up vote 9 down vote

If your program doesn't need to write any OTHER files that would be larger than this limit, you can inform the kernel of this limit using ulimit. Before you run your command, run this to setup a 200MB file size limit for all process run in your current shell session:

ulimit -f $((200*1024))

This will protect your system but it might be jaring for the program writing the file. As eyazici suggests, consider setting up logrotate to prune log files once they reach a certain size or age. You can discard old data or archive it for a period of time in a series of compressed files.

share|improve this answer
this limits the size of any file written by the program – Kim Jul 22 '11 at 10:45
True. If the program had a legitimate need to write other large files, you would need a different solution. – Caleb Jul 22 '11 at 10:46
up vote 5 down vote

You can truncate the output with head:

size=$((200*1024*1024-$(stat -c %s myprogram.log)))
run_program | head -c ${size} >> myprogram.log
share|improve this answer
Very creative. Just a note that this would only work to limit NEW data being written to the file, it would not take into account how large or small the file already was. – Caleb Jul 22 '11 at 11:15
2  
Note that it is likely that this will kill the program (with SIGPIPE) once it has reached the size limit, rather than discarding the data. – Random832 Jul 22 '11 at 11:38
1  
I was thinking similar with some dd magic but yeah @Random832 is right, you'll get a SIGPIPE as head/dd/whatever drops it. – Aaron D. Marasco Jul 23 '11 at 13:07
What about ignoring it with a trap '' SIGPIPE ? – tuxce Jul 25 '11 at 15:35
Or pipe instead to { head -c "$size" >> log; cat > /dev/null; }. – Stephane Chazelas Jan 22 at 11:10
up vote 4 down vote

You may create a new filesystem image, mount it using loop device and put the log file on that filesystem:

dd if=/dev/zero of=./200mb.img bs=1024 count=200000 # create new empty 200MB file
mkfs.ext2 200mb.img # or ext3, or whatever fits your needs
mkdir logs
sudo mount -t ext2 -o loop 200mb.img logs # only root can do '-o loop' by default
run_program >>logs/myprogram.log

You may also use tmpfs instead of a file, if you have enough memory.

share|improve this answer
Creative idea... that leaves it up to the program as to what to do when it runs out. – Aaron D. Marasco Jul 23 '11 at 13:05
up vote 0 down vote

Since it is text, I would write a script in your favorite language and pipe it to that. Have it handle the file I/O (or keep it all in memory and then dump it on SIGHUP or similar). For that, instead of 200MB I would think of a 'reasonable' number of lines to keep track of.

share|improve this answer
Keeping 200MB of log data in memory for no other reason than to be able to truncate it isn't a very good use of system resources. Nor is doing a line count on a big log file. I would recommend using tools built for this like syslog and logrotate. – Caleb Jul 25 '11 at 8:45

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.