I have what may sound like a strange situation. I have a mount point for a directory containing a variety of log files. From time to time it is necessary to unmount this directory and remount it in a different place. (You can think of this as another way of doing
logrotate, although that's not the primary intent.)
The problem is that this is a live system, and there are always log files open. Before doing the unmount and remount, I can try to first stop all processes that are writing, and restart them after the remount, but in my situation I can't always guarantee this. The processes that might be logging aren't all under my control, and there may always be a new one that my mount-switching routine hasn't heard of. Or someone may have interactively logged in and changed to the logging directory. (Also some processes can't be stopped without potentially destabilizing things.) But any process which is writing to a file in the logging directory, or has the logging directory as its current directory, thwarts the unmount, and thus prevents my switchover from working at all.
So what I'm thinking of doing is trying to use a hypothetical "dynamic mount-switching filesystem". This would basically be a simple pass-through virtual filesystem which mapped all filesystem operations through to an underlying real filesystem, but with the ability to dynamically redirect the mapping to a different underlying real filesystem at any time, even with files open.
Naturally there would be some semantic implications during a switchover:
A file open for writing would have to be split, with the previously-written portion left on the old virtual mount point and anything newly written going to the new one. This would work best for files opened in O_APPEND mode, and in fact seeks on written files might have to be disallowed. But all of these consequences would be perfectly fine for sequentially-written log files (which are of course precisely what my application involves).
A file open for reading would have to return EOF or perhaps EIO.
A process with the log directory as its current directory shouldn't be affected (although
lsbefore and after the switchover might yield wildly different results).
Subdirectories of the logging directory might require more thought, but would basically follow the same rules.
Hopefully this all makes sense so far (although you may think I'm crazy for contemplating it).
So my questions are:
Does a virtual filesystem module that will do this already exist? (I know about "union" filesystems and
autofs, which are kind of similar in some ways.)
If no such module exists, have I overlooked any unsurmountable difficulties if I were to try to write one?
Is there a better way to do what I'm trying to do? Remember, my requirement is to do the equivalent of unmounting and remounting a filesystem, but my constraint is that there may well be files open.
(Some of you are still saying, "This is crazy. You just have make sure there are no files open when you do the unmount." And what I am saying is that I simply can not guarantee that. Identifying all processes that might be writing log files is an open-ended problem; any solution that requires identifying them all in advance is totally not future proof. That's why I'm seriously contemplating this crazy dynamic mounting VFS -- I believe it would actually be less work, and more stable.)
Addendum: it's been suggested that I switch all my logging over to syslog (or some other unique, central service) which would then be easily and centrally manageable. But as I said, I don't have control over all processes that are or might be doing logging. For example, one of them is
chrony, which logs things like its tracking information using fwrite, not syslog. (I can't afford to stop and restart chrony, because I don't want to disrupt timing, so for now I'm having to forego most chrony logging.)