I have a script providing login continuously( like 2-3 pages in a second). I was using below one liner.

sh myscript.sh > /home/user/mylog

I am running another script to zip this file when it reaches 100MB and create new file.

gzip /home/user/mylog;touch /home/user/mylog

But I think I am losing some logs in the time between gzip and touch commands.Can somebody suggest a best practice NOT to lose any logs?

migrated from serverfault.com Dec 9 '16 at 14:51

This question came from our site for system and network administrators.

  • In logrotate, there is an option called copytruncate. There are also multiple options around compression. You would want to read up on those. – Aaron Dec 8 '16 at 18:54
  • The man page says:: Note that there is a very small time slice between copying the file and truncating it, so some logging data might be lost. – prado Dec 9 '16 at 9:48
  • serverfault.com/questions/189477/… – Iain Dec 9 '16 at 19:59

Rather than zip the file as its being written to, move it out of the way then zip the static file...

mv /home/user/mylog /home/user/mylog.0; gzip /home/user/mylog.0

You shouldn't need the touch, as the script runs, if the log file disappears, it should create it on its own.

Here is a test: Run this script... for (( ; ; )); do echo "123" >> test.log; sleep 1; done

then move the file to a new name.... mv test.log test.log.0

The new test.log is created automatically, assuming the script is still running.

Try this in the While loop. It worked when I tested it.

while true;do
  exec 3>&1 1>>time.logger 2>&1
  #other stuff...
  • I am not sure that will work, didn't work for me. After moving the file, when I tried to head the file, it show: head: cannot open '/home/user/mylog' for reading: No such file or directory – prado Dec 9 '16 at 9:45
  • you cannot do a "head" on a file that doesn't exist, but the output stream from your script will create the new log file for you. I updated my original answer with an example. – Jeff W. Dec 9 '16 at 12:47
  • Okay. for loop is working, but I was trying with while loop and not working. Any Idea why? giv@8:~$ cat time-loop while true;do date;done giv@8:~$ cat time-loop while true;do date;done giv@8:~$ sh time-loop >> time.logger & [2] 26536 giv@8:~$ head -1 time.logger Fri Dec 9 18:44:06 IST 2016 giv@8:~$ mv time.logger time.logger.0 giv@8:~$ head -1 time.logger head: cannot open 'time.logger' for reading: No such file or directory giv@8:~$ ps aux | grep time-loop| grep -v grep giv 26536 12.5 0.0 4508 1708 pts/11 S 18:44 0:03 sh time-loop – prado Dec 9 '16 at 13:18
  • what is the significance of doing the 'head -1 time.logger' immediately after moving the log? Can't you just get rid of it? can you post your script in its entirety? – Jeff W. Dec 9 '16 at 13:24
  • I tried your for loop example and it works. My issue is with the while loop. The loop I am trying now is while true;do date;done. I put it in a file and run like sh time-loop >> time.logger &. But once I move the time.logger, it is not creating again. I am using head just to check whether the file is created or not, anyway the file is NOT creating again. Please try for yourself. – prado Dec 9 '16 at 13:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.