8

Our application writes data to disk as a huge ring buffer (30 to 150TB); writing new files while deleting old files. As such, by definition, the disk is always "near full".

The writer process creates various files at a net input speed of about 100-150 Mbits/s. Data files are a mixture of 1GB 'data' files and several smaller meta data files. (The input speed is constant, but note new file sets are created only once per two minutes).

There is a separate deleter process which deletes the "oldest" files every 30s. It keeps deleting until there it reaches 15GB free space headroom on the disk.

So in stable operation, all data partitions have only 15GB free space.

On this SO question relating to file system slowdown, DepressedDaniel commented:

Sync hanging just means the filesystem is working hard to save the latest operations consistently. It is most certainly trying to shuffle data around on the disk in that time. I don't know the details, but I'm pretty sure if your filesystem is heavily fragmented, ext4 will try to do something about that. And that can't be good if the filesystem is nearly 100% full. The only reasonable way to utilize a filesystem at near 100% of capacity is to statically initialize it with some files and then overwrite those same files in place (to avoid fragmenting). Probably works best with ext2/3.

Is ext4 a bad choice for this application? Since we are running live, what tuning can be done to ext4 to avoid fragmentation, slow downs, or other performance limitations? Changing from ext4 would be quite difficult...

(and re-writing statically created files means rewriting the entire application)

Thanks!

EDIT I

The server has 50 to 100 TB of disks attached (24 drives). The Areca RAID controller manages the 24 drives as a RAID-6 raid set.

From there we divide into several partitions/volumes, with each volume being 5 to 10TB. So the size of any one volume is not huge.

The "writer" process finds the first volume with "enough" space and writes a file there. After the file is written the process is repeated.

For a brand new machine, the volumes are filled up in order. If all volumes are "full" then the "deleter" process starts deleting the oldest files until "enough" space is available.

Over a long time, because of the action of other processes, the time sequence of files becomes randomly distributed across all volumes.

EDIT II

Running fsck shows very low fragmentation: 1 - 2%. However, in the meantime, slow filesystem access has been traced to various system calls like fclose(), fwrite(), ftello() etc taking a very long time to execute (5 to 60 seconds!).

So far no solution on this problem. See more details at this SO question: How to debug very slow (200 sec) fwrite()/ftello()/fclose()?

I've disabled sysstat and raid-check to see if any improvement.

11
  • 2
    When creating the large data files, do you already use fallocate(fd,FALLOC_FL_ZERO_RANGE,0,length) to allocate the disk space before writing to the file? Could you use a "fixed" allocation size for the large data files (assuming they don't have much variation in size)? This is a difficult case, because the smaller metadata files may cause fragmentation of the large files. Could you use different partitions for the large data files and small metadata files? Dec 5, 2016 at 13:35
  • Do you have any reader processes? Do they read the oldest data files, or is it random? Dec 5, 2016 at 22:15
  • All files are opened with fopen() and no pre-allocation is done. Using different partitions would be difficult. For the large files I could pre-allocate using a heuristic guess of the size. But the final size could be different. Would the allocated space be returned to "free" after fclose() ?
    – Danny
    Dec 6, 2016 at 5:32
  • Mark, yes there are reader processes. The 'deleter' reads directory information and some of the meta-data files. Also, the big data files could be read by the player app. (application is similar to a video server, with constant bit rate in for the recorder and and (if activated) constant bit rate out for the player.
    – Danny
    Dec 6, 2016 at 5:34
  • 1) IMO it would be better if you could make this question self-sufficient. If you were asking a hypothetical question, one answer would be to test it. But you've tested it and found at least one BIG problem; that's the most important reason you're asking, right? 2) Secondly - I was modelling the algorithms you gave as the only significant IO load on this storage. I'm not sure exactly what I'm supposed to understand from the edit mentioning other processes which cause a different distribution of files.
    – sourcejedi
    Jan 1, 2019 at 11:36

3 Answers 3

3

In principle, I don't see why strict ring-buffer writes would pose any challenge regarding fragmentation. It seems like it would be straightforward. The quote sounds to me like it is based on advice from more general write workloads. But looking at the linked SO question I see you have a real problem...

Since you are concerned about fragmentation, you should consider how to measure it! e4defrag exists. It has only two options. -c only shows the current state and does not defrag. -v shows per-file statistics. All combinations of options are valid (including no options). Although it does not provide any explicit method to limit the performance impact on a running system, e4defrag supports being run on individual files, so you can rate-limit it yourself.

(XFS also has a defrag tool, though I haven't used it.)

e2freefrag can show free space fragmentation. If you use the CFQ IO scheduler, then you can run it with a reduced IO priority using ionice.

The quote guesses wrong, the reply by Stephen Kitt is correct. ext4 does not perform any automatic defragmentation. It does not try to "shuffle around" data which has already been written.

Discarding this strange misconception leaves no reason to suggest "ext2/ext3". Apart from anything else, the ext3 code does not exist in current kernels. The ext4 code is used to mount ext3. ext3 is a subset of ext4. In particular when you are creating relatively large files, it just seems silly not to use extents, and those are an ext4-specific feature.

I believe "hanging" is more often associated with the journal. See e.g. comments from (the in-progress filesystem) bcachefs -

Tail latency has been the bane of ext4 users for many years - dependencies in the journalling code and elsewhere can lead to 30+ second latencies on simple operations (e.g. unlinks) on multithreaded workloads. No one seems to know how to fix them.

In bcachefs, the only reason a thread blocks on IO is because it explicitly asked to (an uncached read or an fsync operation), or resource exhaustion - full stop. Locks that would block foreground operations are never held while doing IO. While bcachefs isn't a realtime filesystem today (it lacks e.g. realtime scheduling for IO), it very conceivably could be one day.

Don't ask me to interpret the extent to which using XFS can avoid the above problem. I don't know. But if you were considering testing an alternative filesystem setup, XFS is the first thing I would try.

I'm struggling to find much information about the effects of disabling journalling on ext4. At least it doesn't seem to be one of the common options considered when tuning performance.

I'm not sure why you're using sys_sync(). It's usually better avoided (see e.g. here). I'm not sure that really explains your problem, but it seems an unfortunate thing to come across when trying to narrow this down.

2

Here's an alternate approach, however it's somewhat involved.

Create many smaller partitions, let's say 10 or 20 of them. LVM2 might come in handy in this scenario. Then use the partitions in a ring-buffer fashion as follows:

One of the partitions would always be the 'active' one, where new data gets written to until it is completely full or nearly so. You don't need to leave any headroom. When the active partition has become full or doesn't have enough free space to hold the next chunk of data, switch to the next partition which then becomes the active one.

Your deleter process will always make sure that there is at least one completely empty partition available. If there isn't one--and this is the crucial part--it will simply reformat the oldest partition, creating a fresh new file system. This new partition will later be able to receive new data with minimal to no fragmentation.

3
  • I didn't mention in the question, but that's actually what we do. See the edited question above.
    – Danny
    Jan 1, 2019 at 3:22
  • @Danny if "the time sequence of files becomes randomly distributed across all volumes", then surely you cannot actually do "and this is the crucial part--simply reformat the oldest partition, creating a fresh new file system. This new partition will later be able to receive new data with minimal to no fragmentation."
    – sourcejedi
    Jan 1, 2019 at 14:16
  • Sorry, my bad. Somehow didn't see/read your last two paragraphs. We have 10-12 smaller partitions, but the deleter removes only the oldest files (1GB each) until "enough" free space is available. Then it stops and waits for the disk to be "too full" again. "enough" and "too full" can be adjusted for tuning.
    – Danny
    Jan 2, 2019 at 7:36
0

The problem is almost certainly caused by the ext4 delalloc (Delayed Allocation) default ext4 mount option. It causes the decision of where to write new files to be delayed until sync (either explicit or the implicit sync running periodically) : If the file system is full this operation can involve moving existing files around on disk to create contiguous space for the new file.

Your problem can be solved by adding nodelalloc to your mount options : this will force ext4 to create space when the original write happens (and if necessary to move around existing files to make space). It will slow down the original write if the file system is full and make it appear that it cannot use the buffer cache for writes, but this is preferable to delaying the problem until you have to do sync, since data stuck for long periods in the buffer cache risks being lost if there is a power outage.

Normally delalloc is preferable since it minimizes fragmentation by deciding where to place files only after the full size of new files being written is known. However ext4 does a good job minimizing fragmentation even with nodelalloc, since it will try to pick a large space ahead of time, whenever possible.

1
  • Thanks, that is a great theory. Will test it out. In the meantime we have modified the application to first estimate the max file system and use fallocate() to reserve the space (it then calls truncate() on the file when closing).
    – Danny
    Sep 16, 2021 at 15:10

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .