cat /dev/null > file.log does not truncate large file in Darwin

Question

In the past, on linux systems, I've been able to truncate large, open log files (that is, a file that is being actively written to by a process) using cat /dev/null > file.log.

However, on 10.9 (Mavericks), that doesn't seem to be the case. I've got an 11GB file that is being logged to by an application, but when I perform the same command with said file, nothing seems to happen.

When I try this on a file of trivial size, it does work.

Here is ls -l /dev/null:

crw-rw-rw- 1 root wheel 3, 2 Dec 16 12:49 /dev/null

I've also tried cp /dev/null file.log to no avail.

Thinking that I might take advantage of the truncate function (man 2 truncate in Darwin) I compiled this and ran it against two files, one of trivial size and the other the actual log file. Again, it worked against the trivial file and did not work on the much larger log.

/*
 * Copyright (c) 2013 Thomas de Grivel <thomas@lowh.net>
 *
 * Permission to use, copy, modify, and distribute this software for any
 * purpose with or without fee is hereby granted, provided that the above
 ...
 * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
 */

#include <unistd.h>

int main (int argc, const char **argv)
{
        int e = 0;
        while (--argc) {
                argv++;
                if (truncate(*argv, 0)) {
                        e = 4;
                        warn("%s", *argv);
                }
        }
        return e;
}

The process returns 0 regardless of which file I use.

Answer 1

cat /dev/null is a bit convoluted a way to write a command that produces no output. : or true are more obvious ones.

In all of cat /dev/null > file, : > file, and even > file in most shells, the shell opens file with O_TRUNC on stdout, then runs the application which doesn't output anything, then the file is closed and left truncated.

However, in that case or when using the truncate system call, if the process that is filling up that file didn't open it with the O_APPEND flag, the next time it writes to the file descriptor it has open on the file, it will write the data at the offset it was within the file.

Because HFS+ doesn't support sparse files, that means that the space before that offset will have to be reallocated and filled with zeros by the system.

So, you need to kill the application that is writing to that file before truncating it. Or you need to make sure the application opens the file with O_APPEND (like with >> if using shell redirection).

If you want to experiment with it:

$ exec 3> x
$ yes | head -n 50000 >&3
$ ls -ls x
200 -rw-r--r--  1 me me  100000 Dec 16 21:32 x

Now the fd 3 of my shell is 100000 bytes within the file

$ : > x
$ ls -ls x
0 -rw-r--r--  1 me me  0 Dec 16 21:34 x

Now the file is truncated (size 0, no space used on disk).

$ echo >&3
$ ls -ls x
200 -rw-r--r--  1 me me  100001 Dec 16 21:34 x

Writing 1 byte to the file at offset 100000, the file is now 100001 bytes large, the first ones all zeros, would use over 100k on HFS+, but about just one disk block in most other Unix file systems

On the other hand, with:

$ exec 3>> x
$ yes | head -n 50000 >&3
$ ls -ls x
200 -rw-r--r--  1 me me  100000 Dec 16 21:35 x
$ : > x
$ echo >&3
$ ls -ls x
8 -rw-r--r--  1 me me  1 Dec 16 21:36 x

Writing 1 byte to the file not at offset 100000, but at the end of the file because of O_APPEND. The file is 1 byte large, and takes the space needed to hold that one byte.