unix file is corrupted [closed]

Question

I discovered that one of my logs consumes lots of space which comes from a running process. I wanted to clean this file, so I can run logadm to rotate it after. but I don't know how

# >MyLog_nohup.out
# ls -lLh MyLog_nohup.out
-rw-r-lr--   1 user     group       72G Jul 30 07:26 MyLog_nohup.out
# du -sh MyLog_nohup.out
480K   MyLog_nohup.out

Even after freeing it, still consumes 72G, and running more on it, just showing blank lines.. How can I fix that?

I can't afford restarting the process. But I wanted to use logadm to rotate this log file, is this possible? I tried, but it keeps coping blank lines an infinite loop. same as when I do more on this file. Is there any other way to fix this?

For logrotate, there is copytruncate option that deals with the open file, but I can't use it while the file has these blank lines as it runs in a loop. I still don't understand why I can not view/more/head this file!

Answer 1

Your file is not corrupted; On Linux and POSIX systems, as long as a running process has an opened file descriptor to some file that it is writing, it will be able to continue writing it, even if you remove or rename that file (because a file descriptor is related to an i-node, not to a file name). in particular logrotate or logadm -or any sequence of external commands- won't do anything useful about the disk space.

^{I am assuming you are on Linux.}

If your process has pid 1234, you could look into /proc/1234/ in particular list the /proc/1234/fd/ directory. Read proc(5).

You probably should stop the offending process (using kill -TERM then kill -QUIT then at last kill -KILL; see signal(7) & kill(1)), then remove the file, and at last correct and/or configure your program to do some more useful logs, and start it again.

You probably have lost all the computation done by your program. So better stop it ASAP, improve it (perhaps you want some application checkpointing, or persistence, or add some way to have it close, then rename, and then re-open the log file), and restart an improved version of your program.

You should read Advanced Linux Programming. You probably have several bugs in your program (perhaps related to logging). You might use strace(1) to understand the syscalls done by your process, and you could use syslog(3) inside your (improved) program.

Very probably, you have a design bug in your program. So better stop it now, think, improve it, and start again. Waiting for the disk to be entirely filled won't help you (and it would make the matter worse).

For future testing purposes, you might consider setting some disk quotas and or some resource limits (e.g. setrlimit(2), and bash ulimit builtin).

In the future, always design your program to be able to afford restarting the process. Not being able to afford that is always a huge mistake (in particular, you need some backup strategy, and you need some revision control on your source code; I recommend git for that).

Answer 2

# ls -lLh MyLog_nohup.out
-rw-r-lr--   1 user     group       72G Jul 30 07:26 MyLog_nohup.out
# du -sh MyLog_nohup.out
480K   MyLog_nohup.out

This looks like a sparse file to me. A sparse file allocates a virtual size on the disk (72G in your case) for efficiency but in reality the used space is as much data that is written on the file (480K in your case). The virtual size it's not calculated in the used space (you'll not free 72G in case you delete the file but only 480k).

You see blank lines when reading the file because... well there's only blank lines (nulls) in it. Try hexdump (1) to see the actual data (if any) but for any meaningful output you'll need a tool that understands the format of the file (the program creating the file should come with a tool for that).

Furthermore - as other have pointed out - you can't reclaim the resources of the file while the file remains mapped in memory. You need to free the mapping, that is ether terminate the program or you could try this with gdb (1)

pidof <procname>       # find the pid of the process which creates the log file
ls -l /proc/<pid>/fd/  # find the fd pointing to MyLog_nohup.out
gdb -p <pid>           # attach to pid
(gdb) p close(<fd>)    # close fd pointing to log file

However please note that the program might recreate the log file or have an unspecified behaviour when the fd closes and since you 'can't afford restarting the process' the above might be just too risky, just let the log file be.