1

My server broke down with exenstive damages when a log file reached 27 gigs in a few hours. Log files are compressed daily, and are usually very small, maybe up to 10mb in 24 hours. Today there was an issue, and it caused a stack trace to be printed 20 times a second, I was sleeping, the log turned into 27gb, and when hard drive became full, serious damages and data loss occured. The log file will always be called "latest.log" I need a cron job that will delete this file if it becomes larger then 1GB to prevent this train wreck in the future. Thanks for helping me.

  • can you be more specific? Some frameworks allow you to define rotation and/or a max log size. – Rui F Ribeiro Nov 15 '15 at 10:37
  • 1
    you can limit a process's max file size with ulimit -f. – meuh Nov 15 '15 at 10:40
  • but then the process does not hangs, meuh? – Rui F Ribeiro Nov 15 '15 at 10:44
  • Anything that prints such messages without limiting rate (or just printing the first few before giving up the chatter, or just quitting if a serious problem repeats too often) is suspicious of not-up-to-snuff programming... – vonbrand Nov 15 '15 at 13:43
0

I would not do it with a cron, albeit if you insist in that, a simple line of find would do that. Be aware that you would need to restart the service, because files in Unix only die when they are not being used anymore.

Back in crontab:

*/10 * * * * find /dir -name latest.log -size +1GB -exec rm -f {} \; -exec command_to_restart_your_service \;

As you talk about stacks, I assume you are talking about tomcat. Have a look at the following post. Nevertheless, if that server is so important, I would forward all the logs to a remote log server. Why are you having damages, are you running an SQL server there too? I would run it in a separate server.

Here is the link to the article about limiting logs in size.

https://stackoverflow.com/questions/8342336/how-to-set-maximim-number-of-rolls-and-maximum-log-size-for-tomcat

  • No, it's latest.log from java minecraft. You said you didn't want me to use a cron job, what do you suggest instead? I'm very inexperienced and I welcome any help or advice. Do I just put "find /home/minecraft/multicraft/servers/server1/ -name latest.log -size +1GB -exec rm -f {} \" in /etc/rc.local? – user2656801 Nov 15 '15 at 11:09
  • I edit it out for running every 10 minutes from cron. Mind you the cron must be the user which is running java, or root. I did not put such a long dir name just for clarity. – Rui F Ribeiro Nov 15 '15 at 11:15
  • 4
    deleting the file does not help if a process still has it open. even truncating it will not help (the nameless file keeps on growing as writes happen). you need to restart whatever is logging there to get it to re-open that file (after you delete it). in unix/linux, removing a file only remove the name. the file only goes away once there are zero names and zero opens. – Skaperen Nov 15 '15 at 11:24
  • you should be careful with running -exec rm -f it will automatically delete any files the find algorithm matches. you should at least test the command first by replacing -exec rm-f with -exec ls -l – the_velour_fog Nov 15 '15 at 11:25
  • 2
    I would look into using logrotate - managing log files when they hit a pretermined size or date is exactly what its designed for. Or as @RuiFRibeiro suggests - getting the logs off the server is a good idea - you could do that with rsyslog – the_velour_fog Nov 15 '15 at 11:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.