1

Some file-systems notably XFS and btrfs support Copy on Write at block level for files. This is done by reflinking where the underlying blocks are shared between files until they are modified.

Since a directory is essentially an associative array mapping file names to inodes it should be straight forward to do something similar for directories.

Have any filesystems been developed which can support this on Linux (or any other Unix-like system)?

Presumably it would need kernel support just like use reflinking does. That is a call like copy_file_range() which works with directories.

Is anyone actively working on this? Is it simply that no-one has wanted to do it yet or is there any reason why this is a bad idea or unecessary?

Are there any particular technical obstacles that need to be overcome?

See also https://serverfault.com/questions/129969/is-there-a-way-to-create-a-copy-on-write-copy-of-a-directory which does not really answer this question.

4
  • A directory is a file (that holds the mapping). So XFS and BTRFS already support what your asking for. May 16 at 0:31
  • but copy_file_range returns EISDIR if either file is a directory suggesting it might not be that simple. May 16 at 1:28
  • open() also returns EISDIR but that does not mean you cannot open a directory and get a normal file descriptor with opendir(). Anyway, copy_file_range does not deal with block level I/O, it just gives the underlying filesystem the possibility to implement copy-on-write. I'm not sure what your use case is though, why would you need that for directories. man ioctl_ficlone might help you. May 16 at 1:51
  • This one is more educational than a particular use case. However, if you could reflink whole directories it would be useful for stackoverflow.com/q/72081715/1569204 Coreutils, and in particular cp, does not attempt to reflink directories. Easily proved with by strace cp -rf --reflink=always. May 16 at 7:10

1 Answer 1

3

Are there any particular technical obstacles that need to be overcome?

One big obstacle is the difference in semantics. Copying a directory (in the case where the target doesn’t already exist) results in copies of all its contents, and a new directory pointing to all those contents. In particular, this means that while the source and target directories end up containing the same names, the target inode for each name is different.

If you clone directories in the same way as cloning files, you end up with two directories whose contents point to the same inodes — effectively, you’re creating a directory containing hard links to the original directory’s contents. This won’t work for directories, and it creates surprising semantics for files — cloning dira containing file, as dirb, means that editing dirb/file will also edit dira/file, which presumably wasn’t the original intent.

So this means that sharing storage for directory copies won’t work in the general case, and would only be useful in a small number of cases.

copy_file_range and the FICLONE ioctls however don’t assume that underlying storage will be shared. They are requests of the form “kernel, please copy this (portion of) file”; they have benefits where storage can be shared, but also where delegated copying is more efficient than reading and writing. For example, on networked file systems, cloning can be handled on the server, which is far more efficient than client-driven copies. Applying this reasoning to directory copies, without necessarily sharing underlying storage, could be useful.

Even in scenarios where it’s useful, the implementation and use of the kernel operation would be complex.

File cloning operations have a single file to handle, initially in a single file system. Even then, the extension of copy_file_range to handle copies across multiple file systems has led to a number of issues.

A “clone” operation for directories would be much more complex. Copying a directory involves not only copying the directory itself (i.e. the map of file names to inodes), but also the contents of the directory. As already mentioned, you can’t just hard link files, the semantics would then be different; and even if that were an acceptable simplification, you would still need to deal with entries which can’t be hard linked, such as directories in the directory you’re trying to clone. Then we get to mount points, and the level of “fun” just goes up by a whole lot. All this of course while handling permissions.

Assuming someone implements all this, you then need to update relevant user space programs to use the new operation, with existing code as fallback. Unlike single-file clones, the semantics are rather different, and it may not be easy in all cases to splice the new function in. The resulting file system semantics are also different (arguably, those could be changed without breaking the world if it was worth it — see relatime and noatime).

There are probably other obstacles I haven’t thought of yet; but the first one shows that there is really little opportunity for directories to actually share storage anyway.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.