21

I try to keep last 50 lines in my file where I save temperature every minute. I used this command:

tail -n 50 /home/pi/Documents/test > /home/pi/Documents/test

But the result is empty test file. I thought, it will lists last 50 lines of test file and insert it to test file. When I use this command:

tail -n 50 /home/pi/Documents/test > /home/pi/Documents/test2

it is working fine. There is 50 lines in test2 file.

Can anybody explain me where is the problem?

  • 2
    Something like rrdtool may be more appropriate for keeping N records (among other stats) over time. – thrig Jul 29 '16 at 15:26
  • See also unix.stackexchange.com/a/147620/117549 – Jeff Schaller Jul 29 '16 at 15:27
  • 2
    classic truncation issue – haylem Jul 29 '16 at 15:27
  • If you're using python to generate your logs, you should look into the logging module – Wayne Werner Jul 29 '16 at 16:38
29

The problem is that your shell is setting up the command pipeline before running the commands. It's not a matter of "input and output", it's that the file's content is already gone before tail even runs. It goes something like:

  1. The shell opens the > output file for writing, truncating it
  2. The shell sets up to have file-descriptor 1 (for stdout) be used for that output
  3. The shell executes tail.
  4. tail runs, opens /home/pi/Documents/test and finds nothing there

There are various solutions, but the key is to understand the problem, what's actually going wrong and why.

This will produce what you are looking for,

echo "$(tail -n 50 /home/pi/Documents/test)" > /home/pi/Documents/test

Explanation :

  • $() is called command substitution which executes tail -n 50 /home/pi/Documents/test
  • the quotation marks preserve line breaks in the output.
  • > /home/pi/Documents/test redirects output of echo "$(tail -n 50 /home/pi/Documents/test)" to the same file.
  • Thank you it is works fine! I have one more question. Could you please explain how your procedure works step by step? – dorinand Jul 29 '16 at 15:37
  • 1
    But why in your case bash do not perform > first? I do not understand how bash proccess the command. Can anybody explain? – dorinand Jul 29 '16 at 23:34
  • 1
    The > is on the echo command, so it is executed when the echo command begins execution. It cannot begin execution before it is written. The variable substitution is what writes the command. It runs the nested command and creates the echo command by substituting in the value. – jobermark Jul 30 '16 at 18:12
8

Another solution to the file redirection clearing the file first is to use sponge from the moreutils packge like so:

tail -n 50 /home/pi/Documents/test | sponge /home/pi/Documents/test
6

This is because bash processes the redirection with the > first, deleting the contents of the file. Then it executes the command. Were you to use >>, the last 50 lines would be appended to the end of what's currently in the file. In this case, you'd have the same 50 lines repeated twice.

The command works as expected when redirecting to a different file. Here is one way to write the last 50 lines of a file to a file of the same name:

tail -50 /home/pi/Documents/test > /home/pi/Documents/test2 && mv /home/pi/Documents/test2 /home/pi/Documents/test

This first writes the last 50 lines to a temporary file, which is then moved using mv to replace the original file.

As noted in the comments, this won't work if the file is still open. Moving the file also creates a new inode and may change ownership and permissions. A better way to do this using a temporary file would be:

tail -50 /home/pi/Documents/test > /home/pi/Documents/test2 ; cat /home/pi/Documents/test2 > /home/pi/Documents/test

The temporary file can also be removed, though each time this happens its contents will be overwritten.

  • thank you. Could you please explain me step by step what bash execute ? I cannot imagine how it works. – dorinand Jul 29 '16 at 15:15
  • tail -50 /home/pi/Documents/test >/tmp/foo && cat /tmp/foo >/home/pi/Documents/test – steve Jul 29 '16 at 15:19
  • 1
    note that this won't work if the logfile is still open by the logging process (which will continue logging to the original, deleted file). tempfile + move results in a new inode (breaking any hard-links), and possibly different owner or perms. tail ... > temp ; cat temp > orig ; rm -f temp works. – cas Jul 30 '16 at 5:10
4

Since you've seen the main issue with shell redirection, here's an alternative way to prune a file to its last 50 lines:

file=/path/to/the/file
n=$(( $(wc -l < "$file") - 50 ))
[[ $n -gt 0 ]] && sed -i 1,${n}d "$file"

The hard work is done by (GNU) sed with the -i "in-place editing" feature, which works under the covers by creating the output in a temporary file. The rest of the lines set up the math for sed's operation, namely:

  1. count the lines in the file (wc), then subtract 50; assign that to n.
  2. if n is positive, run the sed command to delete lines 1 through n.
4
printf '%s\n' '1,$-50d'   w | ed -s /home/pi/Documents/tes

printf is used to pipe commands (one per line) into ed. The ed commands are:

  • 1,$-50d -- delete all but last 50 lines
  • w -- write modified file back to disk

There are no redirections involved, so the shell can't overwrite the output file before it has been read.

Also, unlike most forms of "in-place" editing (which typically only simulate "in-place" editing by creating a temp file and then renaming it over the original), ed actually edits the original file - so it keeps the same inode (and owner, group, and permissions - tempfile + mv will always change the inode, and may change the others depending on circumstances).

4

On a slightly different track, you can use logrotate(8) to back up the log files regularly to incrementally named files, and then delete old ones.

This is how the main system log files are managed to prevent them from growing too long.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.