bfq_async_charge_factor says async writes are given 10x less than "fair" share of device throughput. How can I observe this?

Question

I've been reading about CONFIG_WBT, and also BFQ. I tried to compare WBT v.s. CFQ in my hard drive. I learned CFQ tries to control Linux' massive async writeback, but its success is limited because of the hard drive's writeback cache. Disabling the hardware write cache (but leaving NCQ enabled) on my drive allowed much improved control.[1]

[1] Determine the specific benefit of Writeback Throttling (CONFIG_WBT)

I know WBT is nowadays disabled on CFQ/BFQ. Also, since upstream Linux v4.19 is pushing blk-mq as a default for scsi, distributions e.g. Fedora need to switch from CFQ to BFQ by default, or switch back to the "legacy" block layer, or etc., according to their evaluations. So I would like to understand BFQ.

I read BFQ has two hardware-side heuristics. It "overcharges" writes by 10x, to mitigate the effect of device write cache. It also tries to mitigate the effect of NCQ using idling. For now, I am most confused by the write overcharge.

To keep low the ratio between the number of write requests and the number of read requests served, we just added a write (over)charge coefficient: for each sector written, the budget of the active application is decremented by this coefficient instead of one. As shown by our experimental results, a coefficient equal to ten proved effective in guaranteeing high throughput and low latency.

http://algo.ing.unimo.it/people/paolo/disk_sched/bf1-v1-suite-results.pdf

/*
 * Async to sync throughput distribution is controlled as follows:
 * when an async request is served, the entity is charged the number
 * of sectors of the request, multiplied by the factor below
 */
static const int bfq_async_charge_factor = 10;

https://elixir.bootlin.com/linux/v4.18/source/block/bfq-iosched.c#L190

(I don't see any code in BFQ to disable this factor when writeback caching is disabled. I see WBT included some code to track if writeback caching is enabled, for very similar reasons. In principle I assume BFQ could do the same thing, but right now it seems BFQ will always overcharge writes, even though BFQ only claims it is needed on devices with writeback caching).

This says async writes will be given a much lower share of the device throughput. Is there a simple test case to observe this "unfair" share? Or am I mis-understanding?

My link above included a quick test of BFQ. This was a simultaneous read v.s. write with basically default fio settings. I think BFQ gave the reader and writer something much closer to a "fair" share. (The reader achieved 40MB/s on my hard drive).

Answer 1

The overcharge factor has been reduced in v4.19, from 10 to 3. So the current overcharge factor does not raise my attention quite so much :-).

[PATCH BUGFIX/IMPROVEMENT 0/4]

While working a little bit on cgroups I/O control, I found two nasty bugs in bfq. They break bandwidth control in simple configurations with one-process groups. These bugs are fixed by the first two patches in this series.

These fixes improved I/O control so much, that I could reduce the write overcharge factor, used by bfq to counter write-induced issues. This reduction is performed by the third patch.

Indeed I found a very marked difference between the v4.18-stable and v4.19-stable kernels, with a test on my laptop hard drive. I watched read v.s. write throughput using vmstat 1, while running a simultaneous read/write test:

fio --ioengine=psync --size=2G --name=read --rw=read --name=write --rw=write 

On a 4.18-stable kernel, the writer finishes first. On v4.19-stable, the reader finishes first. But the dynamics are even more interesting. On v4.19, vmstat 1 mostly shows the reader being granted 2-3 times the throughput share of the writer, with just a few exceptional samples. On v4.18, the throughput fluctuates wildly between the reader and writer.

This is not at all conclusive on its own. But I can't help thinking about the obvious hypotheses. 1) Perhaps the original 10x overcharge included a substantial "fudge factor", compensating what were actually mistakes in BFQ? 2) Perhaps in v4.19 the overcharge factor of 3 explains why the reader is allocated 2-3 times the throughput share of the writer?

I had also posted about this question on the BFQ mailing list, and Paolo Valente tried to give me an answer. He pointed me towards the overcharge change in v4.19. He also said the overcharge factor is about more than just hardware write caches:

In other words, by keeping the low ratio between async writes requests served and read requests served, inside bfq, the final effect is what one expects from a fair scheduler: a fair bandwidth distribution among processes or groups of processes, regardless of what kind of I/O they do. Probably, I should have explained this compensation problem in my comments, right? If you think so, I'll do it.

At any rate, other practical consequences of this internal 'compensation' of bfq are the incomparably better responsiveness results reported, e.g., in Figure 2 and 5 here (and in many other places):

http://algo.ing.unimo.it/people/paolo/disk_sched/results.php

(I'm referring, in particular, to the cases of writes in the background); or the throughput results with writes in the background in Figures 1 and 4 in the same page.

Things get more complicated with random I/O against sequential I/O, but that's a different story.

https://groups.google.com/forum/#!topic/bfq-iosched/yjZDn_HnLIc