15

A script I wrote does something and, at the end, appends some lines to its own logfile. I'd like to keep only the last n lines (say, 1000 lines) of the logfile. This can be done at the end of the script in this way:

tail -n 1000 myscript.log > myscript.log.tmp
mv -f myscript.log.tmp myscript.log

but is there a more clean and elegant solution? Perhaps accomplished via a single command?

  • logrotate is the elegant solution – Ipor Sircer Sep 19 '16 at 15:04
  • 1
    I've thought of it, but the logrotate configuration would be longer than the script itself... – dr01 Sep 19 '16 at 15:05
  • If logrotate is overkill, your solution is about as elegant as it gets. With sed/awk you might be able to do it in one line but not without a temp file internally, so it's probably not more efficient and probably less readable. – kba Sep 19 '16 at 15:11
22

It is possible like this, but as others have said, the safest option is the generation of a new file and then a move of that file to overwrite the original.

The below method loads the lines into BASH, so depending on the number of lines from tail, that's going to affect the memory usage of the local shell to store the content of the log lines.

The below also removes empty lines should they exist at the end of the log file (due to the behaviour of BASH evaluating "$(tail -1000 test.log)") so does not give a truly 100% accurate truncation in all scenarios, but depending on your situation, may be sufficient.

$ wc -l myscript.log
475494 myscript.log

$ echo "$(tail -1000 myscript.log)" > myscript.log

$ wc -l myscript.log
1000 myscript.log
  • Smart. I marked this as the accepted answer as it doesn't require installation of additional tools. I wish I could accept both yours and @John1024's answer. – dr01 Sep 20 '16 at 9:18
  • Your call. I upvoted the sponge solution as I didn't know about it and it is guaranteed not to mess with empty log lines. This solution has the potential to do that, depending on the log file content. – parkamark Sep 20 '16 at 9:28
18

The utility sponge is designed just for this case. If you have it installed, then your two lines can be written:

tail -n 1000 myscript.log | sponge myscript.log

Normally, reading from a file at the same time that you are writing to it is unreliable. sponge solves this by not writing to myscript.log until after tail has finished reading it and terminated the pipe.

Install

To install sponge on a Debian-like system:

apt-get install moreutils

To install sponge on a RHEL/CentOS system, add the EPEL repo and then do:

yum install moreutils

Documentation

From man sponge:

sponge reads standard input and writes it out to the specified file. Unlike a shell redirect, sponge soaks up all its input before writing the output file. This allows constructing pipelines that read from and write to the same file.

  • 2
    +1 Thanks, I did not know sponge. Very useful for all those who learnt the hard way that you cannot do sort importantfile.txt > importantfile.txt :) – dr01 Sep 20 '16 at 9:14
4

definitely "tail + mv" is much better! But for gnu sed we can try

sed -i -e :a -e '$q;N;101,$D;ba' log
3

For the record, with ed you could do something like

ed -s infile <<\IN
0r !tail -n 1000 infile
+1,$d
,p
q
IN

This opens infile and reads in the output of tail -n 1000 infile (i.e. it inserts that output before the 1st line) and then delete from what was initially the 1st line to the end of file. Replace ,p with w to edit the file in-place.
Keep in mind though that ed solutions aren't suitable for large files.

0

What you can do in your script is implement the logic of log rotation. Do all the logging through a function:

log()
{
   ...
}

This function, firstly, does something like:

printf "%s\n" "$*" >> logfile

then, it checks the size of the file or somehow decides that the file requires rotation. At that point, the file logfile.1, if it exists, is removed, the file logfile.0, if it exists, is renamed to logfile.1 and logfile is renamed to logfile.0.

Deciding whether to rotate could be based on a counter maintained in the script itself. When it hits 1000, it is reset to zero.

If always strictly trimming to 1000 lines is a requirement, the script could count the number of lines in the log file when it starts, and initialize the counter accordingly (or if the count already meets or exceeds 1000, do the rotation immediately).

Or you could obtain the size, such as with wc -c logfile and do the rotation based on exceeding a certain size. This way the file never has to be scanned to determine the condition.

0

I did use, instead of mv, the cp command to achieve it that you are able to have some logfiles right in place where a Software is running. Maybe in the different User home dir or in the app dir and do have all logs in one place as hardlinks. If you use the mv command you lose the hard link. If you use the cp command instead you will keep this hard link.

my code is something like:

TMP_FILE="$(mktemp "${TMPFILENAME}.XXX")"

for FILE in "${LOGFILE_DIR}"/* ; do
    tail -n $MAXLINES "${FILE}" > "${TMP_FILE}"
    if [ $(ls -g "${TMP_FILE}" | awk '{print $4}') -lt $(ls -g "${FILE}" | awk '{print $4}') ] ; then
        cp "${TMP_FILE}" "${FILE}"
    fi
done   

So if the files are on the same Filesystem you may give as well some different rights to the users and in the ${LOGFILE_DIR} you modify the length like I do.

If it is the mv command you lose the hardlink between the files and so your second file is not more connected to the first one - maybe placed some where else.

If on the other place you don't allow someone to erase the file your logs stay together and be nice controlled via your own script.

logrotate maybe nicer. But I am happy with this solution.

Don't be disturbed by the "" but in my case there are some files with spaces and other special letters in and If I don't do the "" around or the {} the whole lot doesn't work nice.

For example there is a Dir where older files gets automated zipped into an OLDFILE.zip and everything that gets zipped is as well listed in File .zip_log so the .zip_log is in this Dir as well but in the LOGFILE_DIR I've got with:

ln .zip_log "${LOGFILE_DIR}/USER_ZIP_log"

the equal file as it is a hard link.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.