How to link a log file to /dev/null and restore the space used automatically

Question

I have a logfile which grows huge. The information that I can get from it is negligible.

I want to link it to /dev/null. However, even if it is "deleted" (see below the output of lsof), it consumes all my hard disk space.

I can truncate it using:

: > "/proc/$pid/fd/$fd"
# for instance:
: > "/proc/2456/fd/2"

Unfortunately, there are processes on the system which are paused when the hard disk is full, and must be manually restarted (and I want to avoid the processes to be paused).

Is there a way to truncate automatically the file when it becomes too big (e.g. when it consumes more than 1G?)

lsof output:

program  2456 user    2w   REG    8,5 441433300992     0 21365598 /home/user/file (deleted)

Answer 1

A file's content is only deleted when there is no reference to it anymore. A reference to a file can be a directory entry or an open file handle. When you remove a file (e.g. with the rm command) that a process still has open (here, the process that's logging to it), the file's content remains until the process closes the file.

The most straightforward way to get rid of old logs is to

1. Move the file to another name, e.g. mv foo.log foo.log.old
2. Instruct the process to reopen its log file. If the process has no way to do that, restart it.
3. Delete the now-closed old log file (rm foo.log.old).

The program logrotate automates this mechanism and can be configured as to how many days of old logs to keep around. It can also compress old logs.

For step 2, if you can't restart the program and it doesn't have a way to reopen its log file, you can try forcibly making it reopen the log file with a debugger. However, beware that this can crash the program if it keeps information about the log file that's now become inconsistent. Proof-of-concept (beware that lots of things can go wrong with this; if in doubt, just don't do it):

gdb -n $pid -batch -x /dev/stdin <<EOF
call close(2)
call open("/path/to/foo.log", 1)
EOF

An alternative crude method to free some disk space if you don't care about any of the logs is to truncate the file. The logging process will keep writing at the same position in the file, but the file will become a sparse file. If you read from the file from the beginning, you'll get null bytes, but these null bytes only occupy a few kB on the disk.

dd if=/dev/null of=/path/to/foo.log

Answer 2

Restart the program to free up the deleted file. Investigate logrotate or the like to properly manage the log data, or tune the program to emit fewer log messages.

Answer 3

When you delete a file, you do not actually "delete" it. You unlink it. The end result is that the program that had the log file open can still access the log file until it closes it (which is rare with log files).

Now to correctly fix your problem your going to want to look into log rotation and log filtering.

Log rotation will allow your to archive, compress, and delete based on a set of rules. So for example all entries older then one day get compressed and all entries over 7 days get deleted.

Log filtering is simply reducing the amount of "stuff" that goes into the log. Some programs you implement the filtering on the program side, others on the logger side. If your using syslogd for example, your can tell it to filter out non critical messages on every thing (again for example).

To quickly fix your issue, either restart the service, find out if it responds to a signal SIGUSR1 and SIGHUP are pretty common, and send that, or restart the machine.

Answer 4

If the log file is being created by output to stderr or stdout, and if you can control the way in which the program creating the log file (let's call it bar) is started, start it with bar &>/dev/null, which will redirect all standard outputs or error messages to /dev/null.

Alternatively, (and I know this is really bad practice, but at least it will work) set up an hourly cron job to kill the program creating the log, delete the log file, and start the program back up, provided that program is not essential to system operation. Again provided the program's name is bar, the way to do this would be echo "* 0 0 0 0 \"/bin/killall [offending program]; /bin/rm [offending log]; [command to run program];\" 1>[crontab location]".

Note, the square brackets are used to denote placeholders, and should not be in the command entered.

Again, please note that the second solution is really bad practice, and overall just a bandage fix... so use it at your own risk.