How do you keep only the last n lines of a log file?

Question

A script I wrote does something and, at the end, appends some lines to its own logfile. I'd like to keep only the last n lines (say, 1000 lines) of the logfile. This can be done at the end of the script in this way:

tail -n 1000 myscript.log > myscript.log.tmp
mv -f myscript.log.tmp myscript.log

but is there a more clean and elegant solution? Perhaps accomplished via a single command?

Answer 1

It is possible like this, but as others have said, the safest option is the generation of a new file and then a move of that file to overwrite the original.

The below method loads the lines into BASH, so depending on the number of lines from tail, that's going to affect the memory usage of the local shell to store the content of the log lines.

The below also removes empty lines should they exist at the end of the log file (due to the behaviour of BASH evaluating "$(tail -1000 test.log)") so does not give a truly 100% accurate truncation in all scenarios, but depending on your situation, may be sufficient.

$ wc -l myscript.log
475494 myscript.log

$ echo "$(tail -1000 myscript.log)" > myscript.log

$ wc -l myscript.log
1000 myscript.log

Answer 2

The utility sponge is designed just for this case. If you have it installed, then your two lines can be written:

tail -n 1000 myscript.log | sponge myscript.log

Normally, reading from a file at the same time that you are writing to it is unreliable. sponge solves this by not writing to myscript.log until after tail has finished reading it and terminated the pipe.

Install

To install sponge on a Debian-like system:

apt-get install moreutils

To install sponge on a RHEL/CentOS system, add the EPEL repo and then do:

yum install moreutils

Documentation

From man sponge:

sponge reads standard input and writes it out to the specified file. Unlike a shell redirect, sponge soaks up all its input before writing the output file. This allows constructing pipelines that read from and write to the same file.

Answer 3

definitely "tail + mv" is much better! But for gnu sed we can try

sed -i -e :a -e '$q;N;101,$D;ba' log

Answer 4

For the record, with ed you could do something like

ed -s infile <<\IN
0r !tail -n 1000 infile
+1,$d
,p
q
IN

This opens infile and reads in the output of tail -n 1000 infile (i.e. it inserts that output before the 1st line) and then delete from what was initially the 1st line to the end of file. Replace ,p with w to edit the file in-place.
Keep in mind though that ed solutions aren't suitable for large files.

Answer 5

What you can do in your script is implement the logic of log rotation. Do all the logging through a function:

log()
{
   ...
}

This function, firstly, does something like:

printf "%s\n" "$*" >> logfile

then, it checks the size of the file or somehow decides that the file requires rotation. At that point, the file logfile.1, if it exists, is removed, the file logfile.0, if it exists, is renamed to logfile.1 and logfile is renamed to logfile.0.

Deciding whether to rotate could be based on a counter maintained in the script itself. When it hits 1000, it is reset to zero.

If always strictly trimming to 1000 lines is a requirement, the script could count the number of lines in the log file when it starts, and initialize the counter accordingly (or if the count already meets or exceeds 1000, do the rotation immediately).

Or you could obtain the size, such as with wc -c logfile and do the rotation based on exceeding a certain size. This way the file never has to be scanned to determine the condition.

Answer 6

I did use, instead of mv, the cp command to achieve it that you are able to have some logfiles right in place where a Software is running. Maybe in the different User home dir or in the app dir and do have all logs in one place as hardlinks. If you use the mv command you lose the hard link. If you use the cp command instead you will keep this hard link.

my code is something like:

TMP_FILE="$(mktemp "${TMPFILENAME}.XXX")"

for FILE in "${LOGFILE_DIR}"/* ; do
    tail -n $MAXLINES "${FILE}" > "${TMP_FILE}"
    if [ $(ls -g "${TMP_FILE}" | awk '{print $4}') -lt $(ls -g "${FILE}" | awk '{print $4}') ] ; then
        cp "${TMP_FILE}" "${FILE}"
    fi
done   

So if the files are on the same Filesystem you may give as well some different rights to the users and in the ${LOGFILE_DIR} you modify the length like I do.

If it is the mv command you lose the hardlink between the files and so your second file is not more connected to the first one - maybe placed some where else.

If on the other place you don't allow someone to erase the file your logs stay together and be nice controlled via your own script.

logrotate maybe nicer. But I am happy with this solution.

Don't be disturbed by the "" but in my case there are some files with spaces and other special letters in and If I don't do the "" around or the {} the whole lot doesn't work nice.

For example there is a Dir where older files gets automated zipped into an OLDFILE.zip and everything that gets zipped is as well listed in File .zip_log so the .zip_log is in this Dir as well but in the LOGFILE_DIR I've got with:

ln .zip_log "${LOGFILE_DIR}/USER_ZIP_log"

the equal file as it is a hard link.

Answer 7

If another process is updating the log file and you want to avoid interfering with it, your best bet may be keeping a second file. The second file (let's call it the "archive file") will hold the last n logged lines.

To do this as cleanly as possible, first move the existing log file to a temporary location. Then combine it with the previously saved archive file, and keep up to n lines.

As for the process which is logging, it should just start a new file the next time it writes a log message.

#!/bin/bash

log_dir=/path/to/my/logs
log_filename=my_file.log
lines_to_keep=1000
archive_filename="${log_filename}.001"

work_dir="$(mktemp -d -p "$logdir")" && \
    touch "$log_dir/$archive_filename" && \
    touch "$log_dir/$log_filename" && \
    mv "$log_dir/$archive_filename" "$work_dir/" && \
    mv "$log_dir/$log_filename" "$work_dir/" && \
    cat "$work_dir/$archive_filename" "$work_dir/$log_filename" | \
        tail -n $lines_to_keep > "$log_dir/$archive_filename" && \
    rm "$work_dir/$archive_filename" && \
    rm "$work_dir/$log_filename" && \
    rmdir "$work_dir"

Answer 8

i created a small script which will delete everything from the /var/log/messages except last 4 lines.

# cat remove-range-of-lines.sh

#!/usr/bin/bash

#print total line number
line_count=`awk 'END{print NR}' /var/log/messages`

#exclude last 4 lines
remove_line=`expr $line_count - 4`

#remove everything except last 4 lines
sed -i '1,'$remove_line'd' /var/log/messages

using this script we can always keep latest entry in the /var/log/messages based on our requirement

if you want to keep last 10000 entries in the /var/log/messages below is and the modified script.

#!/usr/bin/bash

#print total line number
line_count=`awk 'END{print NR}' /var/log/messages`

#exclude last 10000 lines
remove_line=`expr $line_count - 10000`

#remove everything except last 10000 lines
sed -i '1,'$remove_line'd' /var/log/messages