I am trying to understand the Linux IO throttling mechanism. This is what I got so far:

By calling the write (fd, data, size) system call, the data will be written into the OS cache as dirty pages. The OS has two ratios, dirty_background_ratio and dirty_ratio to manage the dirty pages. The OS starts background flushes when the dirty pages reach the dirty_background_ratio. The dirty_ratio is the hard limit that the OS prevents to be exceeded.

If the amount of the dirty pages is less than the dirty_background_ratio, the application (that performs IO) will not be delayed by the OS and the data is simply written on the OS cache as dirty pages (freerun). If the dirty pages exceed dirty_background_ratio, the OS performs background flushes and cleans the dirty pages asynchronously. Though, if I'm correct, still no throttling should occur until reaching the set_point = (dirty_ratio + dirty_background_ratio) / 2. To my understanding, the OS tries to keep the dirty ratio around the set_point. This is what I found in the Wu Fengguang's commit of the Linux kernel:

Users will notice that the applications will get throttled once crossing the global (background + dirty)/2 = 15% threshold, and then balanced around 17.5%.

To actually confirm these points, I implemented a simple application in C which generates 65 GB IO data in 1 GB write chunks (using the write system call). At the same time, I measure the bandwidth (throughput, throttled time), the number of dirty pages, and compute the dirty ratio by dividing the number of dirty pages to dirtyable memory = free memory + reclaimable + file cache (according to global_dirtyable_memory).

My system (CPU: Xeon 6138, RAM: 192 GB) runs CentOS Linux 7 (Core) with the kernel version 3.10.0-862.9.1.el7.x86_64. The IO destination is an SSD device with 400 MB/s write bandwidth. Before running the experiment, I call the sync command and wait so that all dirty pages from before are flushed (you see the global dirty ratio is almost zero at the beginning). The experiments are performed on a clean state system (no other dirtier/expensive processes, only OS management related processes running on the system). Here are my measurements:

IO Throughput / Dirty Pages

The x-ax shows the n-th write of 1 GB data. The measurements match quite well with the computed rates (dirty_background_ratio = 10%, set_point = 15%, balanced around 17.5%). But every time I run the experiment (also on other storage devices), I see a reduced bandwidth after reaching the dirty_background_ratio. You can see this on the red area in the picture. According to what I read on different sources (e.g. the point mentioned above by Wu Fengguang), no throttling should occur until reaching set_point. Whereas I see the throttling starts when reaching dirty_background_ratio and stays steady until reaching set_point where the throttling increases very fast and converges to the disk write bandwidth (sync performance, 400 MB/s).

So here is my question: Why do I experience throttled IO before reaching the set_point (red area)? And how does the OS determine the amount of throttling before reaching set_point?

I asked this question a while ago. In the comments, user495217 wrote that this may be the side effects of the background activity of OS. I wanted to know if this is the case, and if so, does anyone know any resources that already discussed this point? or does someone have any suggestions to run an experiment that can prove this? I have measured the number of CPU Instructions and cycles per second during IO, and have not observed any changes that can prove this.

  • Are you looking at the dirty pages for just your process, or the global dirty pages? What you see as early throttling might just be the system slowing your process in fairness to other things on the system, or from another perspective, it isn't early when you consider dirty pages from other things. At peak bandwidth, dirty pages from other things might already be flushed, or might be lost in the noise.
    – user10489
    Oct 2 '21 at 13:05
  • I am looking at the global dirty pages. But I run the process on a clean state (no other dirtier/expensive processes, only OS management related processes running). Another point that makes me certain this has noting to do with other processes is that I experience this effect every time I repeat the experiment exactly at the same time. The same holds also for other storage devices I have (e.g. an HDD disk).
    – Gongotar
    Oct 2 '21 at 13:09
  • Additionally, before each experiment, I call the sync command and wait to flush all dirty pages before running the experiment. As you see, at the beginning of the experiment, the dirty ratio is almost zero.
    – Gongotar
    Oct 2 '21 at 13:13
  • At 10% (dirty_background_ratio) the flushing of dirty pages starts, which indirectly slightly impedes other processes. But then the hard limit is hit and the kernel actively throttles writers. This makes the rate drop from 2.5/2.0 G to 0.4 G. Your graphs look good but lacks hard data.
    – user495217
    Oct 13 '21 at 14:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.