9

My program creates many small short-lived files. They are typically deleted within a second after creation. The files are in an ext4 file system backed by a real hard disk. I know that Linux periodically flushes (pdflush) dirty pages to disk. Since my files are short-lived, most likely they are not cached by pdflush. My question is, does my program cause a lot of disk writes? My concern is my hard disk's life.

Since the files are small, let's assume the sum of their size is smaller than dirty_bytes and dirty_background_bytes.

Ext4 has default journal turned on, i.e. metadata journal. I also want to know whether the metadata or the data is written to disk.

migrated from serverfault.com Aug 21 '13 at 3:21

This question came from our site for system and network administrators.

  • > My program creates many small short-lived files how much is 'a lot'? Are you deleting these files or rewriting files? > I also want to know whether the metadata or the data is written to disk. I believe the default metadata mode is ordered meaning the metadata is committed before the data is written to disk. Of course there are mount options you can add to change this. > My question is, does my program cause a lot of disk writes? this is difficult to respond to considering the information you have provided. Have you considered using tools such as iotop and sysstat to monitor disk IO? – AngryWombat Aug 21 '13 at 3:39
  • ReiserFS is better for tiny files if you actually want them to hit disk ever tmpfs is fine if you don't care – xenoterracide Aug 21 '13 at 10:01
  • Some clarifications: (1). the ext4 file system is not mounted with sync option. You can consider a default installed fedora, debian or ubuntu. You pick one. (2). Each file is about 60KB. (3). About 1000 files are created and deleted per second, but no more than 10 files exist any any time. In other words, the I/O throughput is large but the space occupied is small. – Wu Yongzheng Aug 22 '13 at 6:07
4

A simple experiment using ext4:

Create a 100MB image...

# dd if=/dev/zero of=image bs=1M count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 0.0533049 s, 2.0 GB/s

Make it a loop device...

# losetup -f --show image
/dev/loop0

Make filesystem and mount...

# mkfs.ext4 /dev/loop0
# mount /dev/loop0 /mnt/tmp

Make some kind of run with short lived files. (Change this to any method you prefer.)

for ((x=0; x<1000; x++))
do
    (echo short-lived-content-$x > /mnt/tmp/short-lived-file-$x
     sleep 1
     rm /mnt/tmp/short-lived-file-$x ) &
done

Umount, sync, unloop.

# umount /mnt/tmp
# sync
# losetup -d /dev/loop0

Check the image contents.

# strings image | grep short-lived-file | tail -n 3
short-lived-file-266
short-lived-file-895
short-lived-file-909
# strings image | grep short-lived-content | tail -n 3

In my case it listed all the file names, but none of the file contents. So only the contents were not written.

  • Nice try. Now I'm convinced. I also tried ext2, and got the same result as you. I changed your parallel I/O workload to a sequential one and got one short-lived-file-999 and a 8 short-lived-content-*. Does anyone have any explanation? – Wu Yongzheng Aug 22 '13 at 15:00
  • @msw: edited in case it was unclear. Otherwise please elaborate. – frostschutz Aug 23 '13 at 11:36
  • That's just silly. The files exist concurrently, there was nothing to overwrite, and filesystems don't overwrite deleted file contents as doing so would harm performance. But by all means, use nbd and log the traffic (or similar method of tracing all writes). – frostschutz Aug 23 '13 at 20:33
7

Unless you are talking about a solid-state drive, a high number of disk writes are not going to be the dominant factor in drive longevity.

If you really want to avoid disk writes at all, look into tmpfs,

  • 2
    tmpfs is indeed a good fit in this case, but I still want to know, as a general operating system question, is the data written to disk (unnecessarily)? – Wu Yongzheng Aug 21 '13 at 5:29
  • Your question would need to be far more specific than you can probably formulate to receive a definitive answer. The buffer cache mediates a complicated trade-off between performance and persistence which cannot be answered in the abstract. Using the tools @AngryWombat listed you could measure the actual writes under from your specific application, but there are so many factors that could make it vary from run to run. – msw Aug 21 '13 at 6:05
  • Well, if pdflush comes after the file is deleted. Writing it would be unnecessary. – Wu Yongzheng Aug 21 '13 at 6:11
1

As a general rule, no, they won't be written. This is because the cache flushes dirty pages when one of two conditions are met:

  1. The data is aged out after /proc/sys/vm/dirty_writeback_centisecs, which defaults to 5 seconds.

  2. There is too little memory for the cache to hold the data, more than dirty_ratio dirty pages in the cache ( defaults to 20% ).

So on a system with plenty of free memory and little write traffic aside from your small files that are deleted in less than 5 seconds, the data won't be flushed.

0

Whether short lived files get written to the disk or not depends not only on default behavior of the kernel file cache, but also on details of file system driver implementation and mount options of the said file system. It is possible to configure the system in such a way that everything will always be immediately written down to disk (essentially, DOS-like behavior).

One file system, prominently featuring the behavior you're interested in (so called "delayed allocation") is XFS. With it you can be more or less sure (given no funny configuration options elsewhere) that blocks belonging to just deleted files will be reused in memory, without intermediate disk access. XFS may still want to update its metadata journal (which will be written to disk rather frequently; yet, given that XFS' journal is metadata only, it is small enough to be set on some other, fast device, such as battery backed RAM found on many RAID controllers).

Because of this behavior, it is not uncommon to find completely zeroed out, but otherwise legit looking files (size and other metadata intact) on an XFS file system after a sudden power interruption. Such is a cost of supporting fast "semi-temporary" file operations.

Some theory

In general, a system call accessing a file system ends, rather quickly, in the file system driver defined method (attached to "struct inode_operations" and "struct file_operations" when VFS driver is registered). What happens after that is left solely to discretion of the file system implementation. Typically, something resembling the following approach is used (this simple example is from linux FAT driver):

if (IS_DIRSYNC(dir))
    (void)fat_sync_inode(dir);
else
    mark_inode_dirty(dir);

If file system is mounted in "sync" mode, all changes go to disk immediately (through fat_sync_inode() in this case). Otherwise, the block is marked as "dirty" and stays in the memory cache until flushed at some reasonable opportunity.

Thus, it is impossible to predict the system behavior in respect to transient files without considering file system mount options and inspecting the source code of its implementation (this, of course, mostly applies to all kinds of exotic file systems mostly found in embedded space).

  • Thanks for your answer. It seems ext4 also has delayed allocation. Does that mean my answer is NO? (given no funny configuration options elsewhere). Does that also mean my answer is YES if ext2 is used? – Wu Yongzheng Aug 21 '13 at 16:11
  • I would think that even with ext2 on modern kernel the answer will be NO. This particular issue was discussed a lot and a brief glance at kernel source shows that ext2 driver mostly relies on "default" kernel operations to do its stuff (thus, everything is delayed by the block cache). I suppose, I should update my answer, to include some extra info. – oakad Aug 22 '13 at 4:04
  • My ext4 is obviously not mounted with sync option. I would never do that. – Wu Yongzheng Aug 22 '13 at 5:51
  • When marking an inode dirty, I assume the file system is responsible for marking the corresponding page dirty. Later when the inode is deleted, does the file system clean the dirty page? If not, the data will be flushed to disk unnecessarily. – Wu Yongzheng Aug 22 '13 at 5:55
  • 2
    Unused data blocks are "released", thus they stop being dirty. If you wrote some stuff to file, and then truncated it prior to flush, the junk past the EOF just disappears (sort of). With metadata it may not be that simple because there may be various trade offs regarding the integrity of file system data structures. By the way, it's not obvious from your question that you always expect to be in full control of your platform - most applications usually end up running on machines of unknown configuration, away from developer. – oakad Aug 22 '13 at 6:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.