Many transient files on journaled filesystem?

Question

Writing and deleting many files quickly

I'm going to write many temporary files per second, over a sustained period. Each will be deleted after some milliseconds.

Say we have an image processing software, that works on partial frames from a fast camera, about 2000 per second. Each of them is written to a file, then the file is changed a little, read again, and deleted, in the same couple of milliseconds.

_{This currently runs on a tmpfs filesystem, but let's assume it can not.
And no, I can not "just fix it doing that", it's needed for some legacy integration.}

How's the filesystem coping?

I would like to understand how this is interacting with the filesystem options. Specifically, I can not get my head wrapped around how a filesystem journal interacts with this.

As a naiive idea, one could hope the data does never touch the disk, and even the metadata does not; After creating a file and then deleting it, the data is the same, right? And after writing and deleting the name in the directory, the directory is also the same, right?

Surprising effects

No, wrong.

A directory can represent the same file names in multiple ways, like by reusing one place to store a new file name, or another place.
That means, when there where write operations on a disk block, it may have changed without filesystem level changes, so it needs to be written to the disk.

But can I make sure it's not written once for each of the four operations? And what about the journal? Will my temporary data end up in the journal? Can I prevent that?

As one question: what should I take care of or keep in mind when doing that?

Answer 1

I can tell you what the theory is, but no hands on experience. I read that files get pushed from memory to disk only every few seconds, and files that are removed quickly enough never end up on disk. Not even in the journal.

I am not sure whether a file needs to exist for full 5 seconds to be pushed out or just happen to exist at the moment of checkpointing, which happens every 5 seconds. Probably latter. If so, some files will get sucked up into the disk. This will happen regularly for brief moments.

There are few mount options that I would suggest looking into. You could make those checkpointing moments arbitrarily rare by increasing the interval, but they will still happen. Also deferred allocation has a side effect of keeping data pushed to disk due to reasons I am not able to explain. I do not remember where I read about that.

• commit=nrsec (Sync all data and metadata every nrsec seconds. The default value is 5 seconds. Zero means default.) on Ext3
• delalloc (Deferring block allocation until write-out time.) on Ext4

Suggested solution: use an in memory filesystem. This way you are sure nothing will be end up on disk. Either tmpfs or ramfs, there are some differences like whether it can swap, grow beyond a specified capacity. I know this is basically what you are using right now. It does not have to be /tmp, you can create a new filesystem of such type over any directory.

http://www.thegeekstuff.com/2008/11/overview-of-ramfs-and-tmpfs-on-linux/