Restoring an open file

Question

I have an interesting problem which may or may not have a solution but I'd love to have one if possible:

On Solaris an open log file has been removed, which still continues to be populated while the process is running but is now inaccessible to all the other tools like cat, tail, etc.

Is there any way to restore the entry in the directory to this file while everything keeps running?

Answer 1

This is doable, with some hacking and limitations (you need root privileges).

First find what file descriptor is using the application to write the log file then create a symbolic link in the previous log file location and pointing to the file /proc entry, eg:

ln -s /var/tmp/file.log /proc/12345/fd/3

The first limitation is that if the file was only open for writing by the process, its permission won't allow an unprivileged user to read its content. However, root and users with the file_dac_read privilege won't be affected. Alternatively, you can use a process to copy the file contents with tail like Gilles is suggesting in his comment. eg:

tail -c +1 -f /proc/12345/fd/5 > /var/tmp/file.log

The second issue is that the whole file content will be lost (ln -s) or part of it (tail -c 1 -f) when the process either closes it or exits.

A workaround is to use a program that monitors this event and backup the file before close is actually called.

Likely tools to do the job are dtrace, truss, mdb, or dbx.

Here is a proof of concept using dtrace on Solaris 10.

#!/bin/ksh
#
# This dtrace script is monitoring a file descriptor for a given process
# and copy its content to the given path when the file is closed.
#

pid=${1:?"$0: Usage: pid fd path"}
fd=${2:?}
path=${3:?}
[[ -f $path ]] && { echo "$path exists"; exit 1; }
dtrace -w -n '
syscall::close:entry
/pid=='$pid' && arg0=='$fd'/
{
        stop();
        system("cp /proc/%d/fd/%d %s",pid,arg0,"'"$path"'");
        system("prun %d",pid);
        exit(0);
}'