How keep last 50 lines in logfile

Question

I try to keep last 50 lines in my file where I save temperature every minute. I used this command:

tail -n 50 /home/pi/Documents/test > /home/pi/Documents/test

But the result is empty test file. I thought, it will lists last 50 lines of test file and insert it to test file. When I use this command:

tail -n 50 /home/pi/Documents/test > /home/pi/Documents/test2

it is working fine. There is 50 lines in test2 file.

Can anybody explain me where is the problem?

Answer 1

The problem is that your shell is setting up the command pipeline before running the commands. It's not a matter of "input and output", it's that the file's content is already gone before tail even runs. It goes something like:

1. The shell opens the > output file for writing, truncating it
2. The shell sets up to have file-descriptor 1 (for stdout) be used for that output
3. The shell executes tail.
4. tail runs, opens /home/pi/Documents/test and finds nothing there

There are various solutions, but the key is to understand the problem, what's actually going wrong and why.

This will produce what you are looking for,

echo "$(tail -n 50 /home/pi/Documents/test)" > /home/pi/Documents/test

Explanation :

• $() is called command substitution which executes tail -n 50 /home/pi/Documents/test
• the quotation marks preserve line breaks in the output.
• > /home/pi/Documents/test redirects output of echo "$(tail -n 50 /home/pi/Documents/test)" to the same file.

Answer 2

Another solution to the file redirection clearing the file first is to use sponge from the moreutils packge like so:

tail -n 50 /home/pi/Documents/test | sponge /home/pi/Documents/test

Answer 3

This is because bash processes the redirection with the > first, deleting the contents of the file. Then it executes the command. Were you to use >>, the last 50 lines would be appended to the end of what's currently in the file. In this case, you'd have the same 50 lines repeated twice.

The command works as expected when redirecting to a different file. Here is one way to write the last 50 lines of a file to a file of the same name:

tail -50 /home/pi/Documents/test > /home/pi/Documents/test2 && mv /home/pi/Documents/test2 /home/pi/Documents/test

This first writes the last 50 lines to a temporary file, which is then moved using mv to replace the original file.

As noted in the comments, this won't work if the file is still open. Moving the file also creates a new inode and may change ownership and permissions. A better way to do this using a temporary file would be:

tail -50 /home/pi/Documents/test > /home/pi/Documents/test2 ; cat /home/pi/Documents/test2 > /home/pi/Documents/test

The temporary file can also be removed, though each time this happens its contents will be overwritten.

Answer 4

printf '%s\n' '1,$-50d'   w | ed -s /home/pi/Documents/tes

printf is used to pipe commands (one per line) into ed. The ed commands are:

• 1,$-50d -- delete all but last 50 lines
• w -- write modified file back to disk

There are no redirections involved, so the shell can't overwrite the output file before it has been read.

Also, unlike most forms of "in-place" editing (which typically only simulate "in-place" editing by creating a temp file and then renaming it over the original), ed actually edits the original file - so it keeps the same inode (and owner, group, and permissions - tempfile + mv will always change the inode, and may change the others depending on circumstances).

Answer 5

Since you've seen the main issue with shell redirection, here's an alternative way to prune a file to its last 50 lines:

file=/path/to/the/file
n=$(( $(wc -l < "$file") - 50 ))
[[ $n -gt 0 ]] && sed -i 1,${n}d "$file"

The hard work is done by (GNU) sed with the -i "in-place editing" feature, which works under the covers by creating the output in a temporary file. The rest of the lines set up the math for sed's operation, namely:

1. count the lines in the file (wc), then subtract 50; assign that to n.
2. if n is positive, run the sed command to delete lines 1 through n.

Answer 6

On a slightly different track, you can use logrotate(8) to back up the log files regularly to incrementally named files, and then delete old ones.

This is how the main system log files are managed to prevent them from growing too long.