rewrite existing file so that it gets replaced by new version atomically, only once fully written

Question

I vaguely recall reading somewhere that there used to be, in some Unices a way to open an existing file for writing, with a flag that asked the kernel to use the old version (for other processes accessing it for reading), until the "new" version was fully written (fd closed), from which point the file appeared as the new version.

In another words, other processes either saw the old version, or the new one, never an incompletely-written one.

Can someone knowledgeable point me to a reference?

Answer 1

This reminds me of Allocate On Flush. When a filesystem uses this feature, instead of writing data directly to disk, it subtracts the size of the data to be written from the disk's free space counter, and holds the data in memory until it a sync system call is performed or the kernel decides to flush the dirty buffers.

In this case, if the file is being modified by one process, and gets opened by another process, the latter process will "see" the unmodified (or "old" if you prefer) version of the file.

Of course, the above are theoretical and depend on various factors, and I'd say a bit unpredictable -since you don't know exactly when the kernel is going flush the dirty pages. For instance in Linux (as you can also read in section 15.3 of Understanding the Linux Kernel), the dirty pages get written to disk under the following conditions:

• The page cache gets too full and more pages are needed, or the number of dirty pages becomes too large.

• Too much time has elapsed since a page has stayed dirty.

• A process requests all pending changes of a block device or of a particular file to be flushed; it does this by invoking a sync(), fsync(), or fdatasync() system call.

This feature is known to be implemented in HFS+, XFS, Reiser4, ZFS, Btrfs and ext4 filesystems.

Answer 2

What youre describing sounds exactly like a basic rename to overwrite a file.

When you rename/move one file on top of another, the old file is unlinked. Meaning the file still exists, but it isnt in the filesystem tree any more. Thus old applications will continue to be able to access the file as long as they keep it open. Once all applications have closed the old file, then its actually unallocated on the disk.

The rename system call is an atomic operation. So to do this you would create a new file under a different name, and then call rename to rename the temporary file as the one you want to replace. Since the operation is atomic, there is absolutely no period where the file is missing. It instantly goes from old file to new file.
Note though that the temporary file and the file being replaced have to reside on the mount point.

Answer 3

As Patrick writes, the usual way to do this is to write the new version to a separate file, and when finished rename the new version to the old filename, overwriting it atomically. This second operation is called overwrite-by-rename.

Now, some references:

• ISO C requires rename to be atomic. From the Open Group Base Specifications:

  If the link named by the new argument exists, it shall be removed and old renamed to new. In this case, a link named new shall remain visible to other processes throughout the renaming operation and refer either to the file referred to by new or old before the operation began.

• Old versions of Mac OS X did not have atomic renames; this is reportedly fixed in Lion.

• Btrfs apparently intentionally violates the standard by not guaranteeing atomic renames, for performance reasons. However, overwrite-by-rename is still atomic, which is all you need for this purpose.