7

I am wondering if there is a file system equivalent to a round-robin database, which for a fixed size, ages off the oldest files. It is pretty easy to implement with a simple cron job, which I have, but I assume it is a problem many people have and there is perhaps something better. I wish to set a fixed-size partition, or pool, in which older files are automatically removed, or aged-off, when the pool is full. A type of circular-buffer that would use the space left by the oldest file for the new ones, whilst preserving file integrity.

My cron solution compares disk usage to a threshold and recursively removes the oldest file until disk usage is again under the threshold. It is not perfect because one can't guarantee the threshold is low enough that it isn't overtaken between two cron iterations. It also doesn't maximize the use of the storage space because of the threshold value which tends to be predictive in nature (how much can I fill in one minute, between two iterations of crond). Two shortcomings I am hoping to improve upon.

I am looking for a more elegant solution, akin to how the round-robin database (http://linux.die.net/man/1/rrdtool) handles this transparently, but for file systems.

|improve this question
  • what about overloading the mv command to perform the verification and deletion (as required) before moving files into the partition? – user153222 Feb 12 '14 at 3:12
  • providing an alternate method of achieving the goal, hopefully triggering some useful comments. – user153222 Feb 12 '14 at 3:30
  • any merit in considering logrotate? I believe you can write rules for any kind of files, but how to guarantee it runs in a timely fashion (when needed) to store more files. – user153222 Feb 12 '14 at 3:33
  • 1
    I disagree. Filesystems are built with various features. Some perform de-deplucation, some allow distributed storage. My need is for one that will maximize the use the fixed size storage it has, and intelligently use the space left by removing the oldest file to store the new ones. – user153222 Feb 12 '14 at 12:05
  • 1
    I agree with user153222, above; tmpfs is an excellent example of a file system that most definitely does not store information in a resilient manner, but is very useful in certain circumstances because of what it does do. You use a tool to serve a need, and I can easily conceive needs which a ring-buffered file system would well serve. I am by no means sure this is a bad question, and I hope it doesn't get closed. – MadHatter supports Monica Feb 12 '14 at 13:20
5

There exist many HSM (Hierarchical Storage Management) systems, mainly aimed at SAN systems. These migrate files from faster disks, to slower disks, to tape as their last-access-time becomes older. You might like to seek out one of these, if you have a SAN. Most of the ones I know of are commercial licenses though, such as the IBM Tivoli HSM that we use. You might like to take a look at OHSM though.

If you just want to delete old files, then a simple cron job such as find /data -atime +30 -exec rm {} \; will delete files that have not been accessed in a certain amount of time (but make sure the filesystem is not mounted with the noatime option!) This would be highly risky, though, unless you had a good online backup system.

|improve this answer
  • this is pretty much what I have now, but not quite sufficient to guarantee the filesystem does not fill between two cron iterations – user153222 Feb 12 '14 at 3:09
  • 5
    I would say that you're doing it the wrong way. Either add a lot of disk to the filesystem, increase the frequency of your cron, or find some other way. If you find yourself needing to delete an unspecified amount of data regularly like this there's probably a problem with whatever is filling the filesystem. I take it that the filesystem in question is a temporary one; in that case, the processes writing to it should be written to clean up after themselves better, and you should probably increase the size of the filesystem. – Steve Shipway Feb 12 '14 at 3:12
  • I appreciate the comment Steve, but I have a legitimate need for a file system that will retain files in a "best effort" kind of way. Under normal conditions, the files are removed periodically by a remote process. In the case of a communications outage, I wish the pool to retain as much as possible, aging off whatever falls off the top, retaining the most recent files up to the capacity of the pool. I am looking for an automated way this can be done, just like it happens in rrdtool. My alternate solutions are ok, but then I need to set a threshold which ensures I do not fill the pool too fast – user153222 Feb 12 '14 at 3:16
  • 1
    So - the filesystem is used a bit like a queue, and when it fills, you'd rather lose the oldest entry than the newest? That makes sense. However, this is really something that should be managed by the application writing to it... I would say that you should (a) increase the FS size so the risk of a problem is lower, and (b) you can probably write a simple perl script to query FS free space, and delete files in reverse order of age until free space > X. This might be harder if the directory structure is more complex though. – Steve Shipway Feb 12 '14 at 3:23
  • Exactly like a queue yes. Cannot increase FS (it is a RAMDISK). I already have the equivalent of the perl script you mention as mentioned above, but am looking for a more elegant solution if one exists. Directory structure is flat in this case, but if such filesystem existed, I would hope the queue could still work even in complex hierarchies. – user153222 Feb 12 '14 at 3:28
0

I can only hope that there is no such file system in existence (cursory searches don't seem to return much of interest). This sounds like the plot for a horrible dream...

Anyway, I presume the short answer is: Nothing like this exists.

And I do dearly hope that is true :(

As for your cron ... if you are only worried about overlapping crons... Use a lock file. Then your only worry is, "Can my application fill this filesystem before cron runs again?"

|improve this answer
  • Not worried about overlapping crons, but am worried about filling the partition before the cron daemon runs again. – user153222 Feb 12 '14 at 3:08
  • horrible dream is not a valid comment in my mind: rrdtool exits for a reason. And imagine you need to store files instead of data in a database... Not that far fetched. – user153222 Feb 12 '14 at 3:10
  • interesting how some people choose to focus on second-guessing my need rather than recommending solutions... rrdtool has the capability to age off older data using a moving window analogy, an retains a fixed size. This is how I wish the partition to behave. I have no need for SQL features. – user153222 Feb 12 '14 at 3:23
  • 4
    The second guessing of need comes from years of people bringing us XY problems. Not nesessarily the case here, but since you dont describe what you are actually doing we tend to assume thats the case for unusual requirements. meta.stackexchange.com/questions/66377/what-is-the-xy-problem – Grant Feb 12 '14 at 4:03
  • Very good, thanks for the link. I don't think it is the case here, I describe my need first, the best I can. I also provide the solution that I have and the two shortcomings I am trying to overcome. I really don't understand why some would flag this with "minimum understanding"-> clearly not the case in my mind. I also disagree with your statement I don't describe what I am doing. I provide the use case in the comments above. – user153222 Feb 12 '14 at 12:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.