0

I have enabled logrotation under springboot directory and it is working but what i see is, "content-data-svc.log" is actually 2MB but when i do ls -ltrh it is showing as 61MB.

if i view the log file, there are more empty lines in the log file and thus creating file size to be large. 50% of the log file is with empty space and rest are log entries. Any idea why this is occurring?

[aemelics@springboot]$ ls -ltrh
total 4.6M
-rw------- 1 aemelics aemelics 1.6M Apr 11 06:44 content-data-svc.log.2.gz
-rw------- 1 aemelics aemelics 1.1M Apr 12 00:44 content-data-svc.log.1.gz
-rw------- 1 aemelics aemelics  61M Apr 12 02:00 content-data-svc.log
[aemelics@springboot]$ du -shx content-data-svc.log
2.0M    content-data-svc.log

Below is my logrotate entries:

[aemelics@springboot]$ cat /etc/logrotate.d/react

/logs/springboot/*.log*
{
su aemelics aemelics
    missingok
    daily
    minsize 20M
    copytruncate
    notifempty
    sharedscripts
    compress
    rotate 5
    postrotate
    endscript
}
4
  • (Probably) unrelated comment: The pattern should be /logs/springboot/*.log (no * at the end), or logrotate will try to rotate the already rotated log files.
    – Kusalananda
    Apr 12, 2020 at 13:00
  • Do you know if the program writing to this log file has opened the file for appending to? If it hasn't, and the logfile is truncated, it would create a "hole" at the start of the file.
    – Kusalananda
    Apr 12, 2020 at 13:02
  • Thanks for the response. Yes, the program keeps on writing the logs into the content-data-svc.log for every second. Will this cause an conflict?
    – Surendar
    Apr 12, 2020 at 13:13
  • if i vi the file - 60% of the file is occupied with [root@wsvmlncdp02 11]# vi content-data-svc.log.1 ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
    – Surendar
    Apr 12, 2020 at 13:39

1 Answer 1

1

The reason that the file looks large in the output of ls but small in the output of du, and that there appears to be empty space at the start of the file after rotation is that whatever program is writing to the log file is not opening the file for "appending" to.

When the log file is rotated, with the copytruncate option set in the logrotate configuration file, a copy of the file is created and the original file is truncated. When a file is truncated, it's contents is effectively deleted, but the file itself is not deleted.

Ordinarily, a program would have opened the file for appending to. This means that each new write to the log happens at the end of the file, always. When the file is truncated, this means that the next log line would be written at the start of the file, because that's also where the end is at that point.

However, if the program hasn't opened the file in append mode, the next write will go to whatever offset into the file the last write finished at, regardless of whether the file was truncated or not.

If the file has been truncated by logrotate, this means that the write creates a hole in the file between the start and the point where the write happens. This hole is full of null bytes, which the vi editor shows as ^@.

This is what happens in your case.

A file with such a hole is called a "sparse" file. The null bytes in the hole itself are not actually stored on disk, which is why du shows a quite small size, but if you read the file from start to finish, you would read 61M of data (the logical size of the file, which is what ls shows).

To solve this, either

  1. rewrite the program that writes to the log file so that it opens the file in append mode, or
  2. tell the program to reopen the log file when the file is rotated (this is usually done by sending the program a HUP signal from logrotate, but you should read your program's manual), or
  3. restart the program from logrotate when the log is rotated.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .