68

I wrote a script that does something and then appends some lines to its own logfile. I'd like to keep only the last n lines (say, 1000 lines) of the logfile. This can be done at the end of the script in this way:

tail -n 1000 myscript.log > myscript.log.tmp
mv -f myscript.log.tmp myscript.log

However, is there a more clean and elegant solution? Perhaps accomplished via a single command?

4
  • logrotate is the elegant solution Commented Sep 19, 2016 at 15:04
  • 3
    I've thought of it, but the logrotate configuration would be longer than the script itself...
    – dr_
    Commented Sep 19, 2016 at 15:05
  • 1
    If logrotate is overkill, your solution is about as elegant as it gets. With sed/awk you might be able to do it in one line but not without a temp file internally, so it's probably not more efficient and probably less readable.
    – kba
    Commented Sep 19, 2016 at 15:11
  • I found better way to get last couple of days log only days1=$(date +%Y-%m-%d -d "1 day ago") days0=$(date +%Y-%m-%d) grep -i "\|$days1\|$days0" myscript.log > myscript.log.new mv myscript.log.new myscript.log Commented Mar 9, 2022 at 13:53

9 Answers 9

78

It is possible like this, but as others have said, the safest option is the generation of a new file and then a move of that file to overwrite the original.

The below method loads the lines into BASH, so depending on the number of lines from tail, that's going to affect the memory usage of the local shell to store the content of the log lines.

The below also removes empty lines should they exist at the end of the log file (due to the behaviour of BASH evaluating "$(tail -1000 test.log)") so does not give a truly 100% accurate truncation in all scenarios, but depending on your situation, may be sufficient.

$ wc -l myscript.log
475494 myscript.log

$ echo "$(tail -1000 myscript.log)" > myscript.log

$ wc -l myscript.log
1000 myscript.log
6
  • Smart. I marked this as the accepted answer as it doesn't require installation of additional tools. I wish I could accept both yours and @John1024's answer.
    – dr_
    Commented Sep 20, 2016 at 9:18
  • 1
    Your call. I upvoted the sponge solution as I didn't know about it and it is guaranteed not to mess with empty log lines. This solution has the potential to do that, depending on the log file content.
    – parkamark
    Commented Sep 20, 2016 at 9:28
  • 2
    This solution has a race condition. If you are unlucky, the redirect -into- the file happens before the reading -from- the file and you end up with an empty file.
    – Coroos
    Commented Jan 3, 2020 at 11:29
  • @Coroos I ran this command for a 6GB file and no issues. Commented Dec 19, 2021 at 8:54
  • 1
    The race condition is between the opening of the file for reading by tail and the piping into the same filename by the shell that executes the command. The size of the file does not matter.
    – Coroos
    Commented Dec 20, 2021 at 21:43
47

The utility sponge is designed just for this case. If you have it installed, then your two lines can be written:

tail -n 1000 myscript.log | sponge myscript.log

Normally, reading from a file at the same time that you are writing to it is unreliable. sponge solves this by not writing to myscript.log until after tail has finished reading it and terminated the pipe.

Install

To install sponge on a Debian-like system:

apt-get install moreutils

To install sponge on a RHEL/CentOS system, add the EPEL repo and then do:

yum install moreutils

Documentation

From man sponge:

sponge reads standard input and writes it out to the specified file. Unlike a shell redirect, sponge soaks up all its input before writing the output file. This allows constructing pipelines that read from and write to the same file.

6
  • 7
    +1 Thanks, I did not know sponge. Very useful for all those who learnt the hard way that you cannot do sort importantfile.txt > importantfile.txt :)
    – dr_
    Commented Sep 20, 2016 at 9:14
  • 3
    Brilliant solution. Worked like a charm. Commented Mar 19, 2020 at 10:11
  • The solution works really well. But the disk is still full. I do not understand what happened. I have a 80Gb drive and one log file grew out of control and was 33Gb. The disk had only 8Gb remaining. I truncated the log file using the above solution and it is now 100Mb. However, the df utility still showing almost the same free disk space of about 8Gb. How is it possible?
    – cha
    Commented Aug 5, 2020 at 22:01
  • 1
    It's possible that the process that writes to the log is still holding on to the original file handle, and so the system things that the storage is still being consumed? Maybe restart that process?
    – Jon V
    Commented Apr 8, 2022 at 14:34
  • 1
    Note that this doesn't preserve file ownership
    – ZuLu
    Commented Jul 14, 2022 at 17:15
6

definitely "tail + mv" is much better! But for gnu sed we can try

sed -i -e :a -e '$q;N;101,$D;ba' log
1
  • 5
    Care to explain whot this magic actually do? Thanks.
    – Eric
    Commented Jul 11, 2021 at 8:14
4

For the record, with ed you could do something like

ed -s infile <<\IN
0r !tail -n 1000 infile
+1,$d
,p
q
IN

This opens infile and reads in the output of tail -n 1000 infile (i.e. it inserts that output before the 1st line) and then delete from what was initially the 1st line to the end of file. Replace ,p with w to edit the file in-place.
Keep in mind though that ed solutions aren't suitable for large files.

1

In the ksh93 shell:

$ seq -f 'This is line %g' 10000 > file.log
$ tail -n 5 file.log 1<>; file.log
$ cat file.log
This is line 9996
This is line 9997
This is line 9998
This is line 9999
This is line 10000

The n<>; file is like the standard n<> file redirection operator which opens the file on fd n in read+write mode without truncation, with the difference that with <>;, if the command being redirected succeeds, ksh93 does an implicit ftruncate() on the fd in place at the position where the command left it.

With other shells, you can do the same with:

{
  tail -n 5 file.log &&
    perl -e 'truncate STDOUT, tell STDOUT'
} 1<> file.log
0

What you can do in your script is implement the logic of log rotation. Do all the logging through a function:

log()
{
   ...
}

This function, firstly, does something like:

printf "%s\n" "$*" >> logfile

then, it checks the size of the file or somehow decides that the file requires rotation. At that point, the file logfile.1, if it exists, is removed, the file logfile.0, if it exists, is renamed to logfile.1 and logfile is renamed to logfile.0.

Deciding whether to rotate could be based on a counter maintained in the script itself. When it hits 1000, it is reset to zero.

If always strictly trimming to 1000 lines is a requirement, the script could count the number of lines in the log file when it starts, and initialize the counter accordingly (or if the count already meets or exceeds 1000, do the rotation immediately).

Or you could obtain the size, such as with wc -c logfile and do the rotation based on exceeding a certain size. This way the file never has to be scanned to determine the condition.

0

I did use, instead of mv, the cp command to achieve it that you are able to have some logfiles right in place where a Software is running. Maybe in the different User home dir or in the app dir and do have all logs in one place as hardlinks. If you use the mv command you lose the hard link. If you use the cp command instead you will keep this hard link.

my code is something like:

TMP_FILE="$(mktemp "${TMPFILENAME}.XXX")"

for FILE in "${LOGFILE_DIR}"/* ; do
    tail -n $MAXLINES "${FILE}" > "${TMP_FILE}"
    if [ $(ls -g "${TMP_FILE}" | awk '{print $4}') -lt $(ls -g "${FILE}" | awk '{print $4}') ] ; then
        cp "${TMP_FILE}" "${FILE}"
    fi
done   

So if the files are on the same Filesystem you may give as well some different rights to the users and in the ${LOGFILE_DIR} you modify the length like I do.

If it is the mv command you lose the hardlink between the files and so your second file is not more connected to the first one - maybe placed some where else.

If on the other place you don't allow someone to erase the file your logs stay together and be nice controlled via your own script.

logrotate maybe nicer. But I am happy with this solution.

Don't be disturbed by the "" but in my case there are some files with spaces and other special letters in and If I don't do the "" around or the {} the whole lot doesn't work nice.

For example there is a Dir where older files gets automated zipped into an OLDFILE.zip and everything that gets zipped is as well listed in File .zip_log so the .zip_log is in this Dir as well but in the LOGFILE_DIR I've got with:

ln .zip_log "${LOGFILE_DIR}/USER_ZIP_log"

the equal file as it is a hard link.

0

If another process is updating the log file and you want to avoid interfering with it, your best bet may be keeping a second file. The second file (let's call it the "archive file") will hold the last n logged lines.

To do this as cleanly as possible, first move the existing log file to a temporary location. Then combine it with the previously saved archive file, and keep up to n lines.

As for the process which is logging, it should just start a new file the next time it writes a log message.

#!/bin/bash

log_dir=/path/to/my/logs
log_filename=my_file.log
lines_to_keep=1000
archive_filename="${log_filename}.001"

work_dir="$(mktemp -d -p "$logdir")" && \
    touch "$log_dir/$archive_filename" && \
    touch "$log_dir/$log_filename" && \
    mv "$log_dir/$archive_filename" "$work_dir/" && \
    mv "$log_dir/$log_filename" "$work_dir/" && \
    cat "$work_dir/$archive_filename" "$work_dir/$log_filename" | \
        tail -n $lines_to_keep > "$log_dir/$archive_filename" && \
    rm "$work_dir/$archive_filename" && \
    rm "$work_dir/$log_filename" && \
    rmdir "$work_dir"
-2

i created a small script which will delete everything from the /var/log/messages except last 4 lines.

# cat remove-range-of-lines.sh

#!/usr/bin/bash

#print total line number
line_count=`awk 'END{print NR}' /var/log/messages`

#exclude last 4 lines
remove_line=`expr $line_count - 4`

#remove everything except last 4 lines
sed -i '1,'$remove_line'd' /var/log/messages

using this script we can always keep latest entry in the /var/log/messages based on our requirement

if you want to keep last 10000 entries in the /var/log/messages below is and the modified script.

#!/usr/bin/bash

#print total line number
line_count=`awk 'END{print NR}' /var/log/messages`

#exclude last 10000 lines
remove_line=`expr $line_count - 10000`

#remove everything except last 10000 lines
sed -i '1,'$remove_line'd' /var/log/messages
3
  • It is not clear from your text in what ways this is better than what the user already uses.
    – Kusalananda
    Commented Aug 12, 2021 at 16:31
  • You wouldn't delete old log messages individually. You would start a new log file and archive the old one. Use logrotate to do this automatically for you. Commented Aug 13, 2021 at 1:04
  • there would be some exception case might come like no free space available to store log archive file (logrotate) in the server. For that kind of situation we have to keep only latest logs and remove other old log entries. This script helped me to achieve my requirement without increase the disk space and keeping latest log entries. Commented Aug 13, 2021 at 6:58

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .