2

The current default behavior of Docker is COW (copy on write), aka allocate on write. This relies on free space in a drive in order to write to disk.

In contrast - with memory, unreferenced files are left available, to be overwritten if something else is needed, or re-linked in constant time if they are needed once again.

We'd like to implement a similar mechanism for caching remote files on a local disk. That is, there would be a set location for files which would be allowed to be overwritten if the space was needed, or linked if the files themselves were needed.

Such a piece of software would ideally hook in to the FS driver when it tries do a write, or when it reports space available. It is my assumption that a polling method would be insufficient, as a piece of software may allocate arbitrary large files at any time.

Does anything like this exist already in the open source world? If not, is it possible? Are there major impediments? What is a good way to get started?

2

A way to get started is with FUSE, filesystem in userspace. This api allows you to implement a filesystem in a small program that can do whatever you choose to cache access to remote files. For example the sshfs fuse filesystem uses sftp to get/put files on a remote system and transparently let them appear as local files.

A readonly caching filesystem in C using FUSE that could form a basis for your code was announced in 2004 and is here. For initial development you might prefer to use a Python library interface to FUSE, and there are at least 3 different implementations, and many example filesystems built on each of them. A search for Python caching FUSE came up with CacheFS from 2011.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.