using VFS to do dynamic mounting

Question

I have what may sound like a strange situation. I have a mount point for a directory containing a variety of log files. From time to time it is necessary to unmount this directory and remount it in a different place. (You can think of this as another way of doing logrotate, although that's not the primary intent.)

The problem is that this is a live system, and there are always log files open. Before doing the unmount and remount, I can try to first stop all processes that are writing, and restart them after the remount, but in my situation I can't always guarantee this. The processes that might be logging aren't all under my control, and there may always be a new one that my mount-switching routine hasn't heard of. Or someone may have interactively logged in and changed to the logging directory. (Also some processes can't be stopped without potentially destabilizing things.) But any process which is writing to a file in the logging directory, or has the logging directory as its current directory, thwarts the unmount, and thus prevents my switchover from working at all.

So what I'm thinking of doing is trying to use a hypothetical "dynamic mount-switching filesystem". This would basically be a simple pass-through virtual filesystem which mapped all filesystem operations through to an underlying real filesystem, but with the ability to dynamically redirect the mapping to a different underlying real filesystem at any time, even with files open.

Naturally there would be some semantic implications during a switchover:

• A file open for writing would have to be split, with the previously-written portion left on the old virtual mount point and anything newly written going to the new one. This would work best for files opened in O_APPEND mode, and in fact seeks on written files might have to be disallowed. But all of these consequences would be perfectly fine for sequentially-written log files (which are of course precisely what my application involves).

• A file open for reading would have to return EOF or perhaps EIO.

• A process with the log directory as its current directory shouldn't be affected (although ls before and after the switchover might yield wildly different results).

• Subdirectories of the logging directory might require more thought, but would basically follow the same rules.

Hopefully this all makes sense so far (although you may think I'm crazy for contemplating it).

So my questions are:

1. Does a virtual filesystem module that will do this already exist? (I know about "union" filesystems and autofs, which are kind of similar in some ways.)

2. If no such module exists, have I overlooked any unsurmountable difficulties if I were to try to write one?

3. Is there a better way to do what I'm trying to do? Remember, my requirement is to do the equivalent of unmounting and remounting a filesystem, but my constraint is that there may well be files open.

(Some of you are still saying, "This is crazy. You just have make sure there are no files open when you do the unmount." And what I am saying is that I simply can not guarantee that. Identifying all processes that might be writing log files is an open-ended problem; any solution that requires identifying them all in advance is totally not future proof. That's why I'm seriously contemplating this crazy dynamic mounting VFS -- I believe it would actually be less work, and more stable.)

---

Addendum: it's been suggested that I switch all my logging over to syslog (or some other unique, central service) which would then be easily and centrally manageable. But as I said, I don't have control over all processes that are or might be doing logging. For example, one of them is chrony, which logs things like its tracking information using fwrite, not syslog. (I can't afford to stop and restart chrony, because I don't want to disrupt timing, so for now I'm having to forego most chrony logging.)

Answer 1

A possible solution could be sending logs to a non-relational database.

You can also try some hack selecting a filesystem that supports writes from multiple sources at the same time.

Both of those two solutions have their own set of problems, and do not appeal to me.

Another possible solution as you lay out the problem, is sharing a common /var/log/machine directory over NFS. It is also a nasty hack that may work with a couple of servers, but that does not scale well.

I would create a central syslog server, and as possible point all the logs to it. In that way, at least in those logs, the problem of having simultaneous is solved.

As for others applications that do not lend themselves to be directly syslog'ed, or even for syslog, I would look at filebeat or graylog.

Ultimately, I would try to have processes the non-syslogable logs over to a central syslog (I was doing the latter with Apache and Tomcat logs).

You can also have a look at other commercial offerings, such as loggly, Scalyr. New Relic or Splunk Enterprise. As an added bonus, you can then have very nice stats to show to management and do capacity planning.

After outlaying and having a well-defined process, you then establish internal rules that whoever needs to install a new solution has to send over the logs to the official destination with the "official" method.

P.S. You are trying to solve a organizational problem with a technical solution. Often that does not go too well. You have to engage your higher-ups.