NOTE: My question stems from this other U&L Q - What exactly is iodepth in fio?

I want to know how internally FIO sets I/O depth. I.e., one of the parameters we submit to FIO when we run it is "IOdepth" (--iodepth=). How does FIO internally control this parameter with the underlying operating system?

Here is an example of the command that we use to run FIO benchmark:

$ sudo fio --filename=/dev/nvme0n1 --direct=1 --rw=randwrite --refill_buffers \
    --norandommap --randrepeat=0 --ioengine=libaio --bs=8K --iodepth=72 --numjobs=256 \
    --time_based --runtime=600 --allow_mounted_write=1 --group_reporting --name=benchtest
benchtest: (g=0): rw=randwrite, bs=8K-8K/8K-8K/8K-8K, ioengine=libaio, iodepth=72

As in this example, the value of "iodepth" can be changed. Accordingly, fio is passing this value to the operating system. So how does FIO do this?

If you want an actual problem to be solved: if I want to write a benchmark program like fio, how would I control the IO queue depth?

  • Welcome on the Unix SE! I tried to make your question more acceptable. Give more context, details. Explain, what you want to know what is not on the another question. – user259412 Aug 5 '18 at 10:42
  • 1
    Please read the response that I've added to this linked Q - unix.stackexchange.com/questions/459045/…. That Q was posted to the Flexible I/O Testers forum and the response looks to cover your Q as well. – slm Aug 5 '18 at 17:46
  • @slm I have checked your comment there. However, I have the same question as one of the guys commented there. If it means number of ios at a given time, then doesn't the numjobs parameter do the same thing. – ahmadi Aug 6 '18 at 1:07
  • @ahmadi - I would ask your Q on that forum and report back here. – slm Aug 6 '18 at 1:09

fio is passing this value [depth] to the operating system


Accordingly, fio is passing this value to the operating system. So how does FIO do this?

There might be a misconception here: fio is NOT directly passing a depth parameter to the operating system. If possible, fio tries to submit I/O up to the iodepth specified using a given ioengine. If that depth is reached then fio will wait for (some) outstanding I/O to complete before it tries to submit more I/O...

How does FIO benchmark set [io]depth?

It depends on the ioengine, it depends on the fio parameters mentioned in What exactly is iodepth in fio? , https://serverfault.com/questions/923487/what-does-iodepth-in-fio-tests-really-mean-is-it-the-queue-depth and https://www.spinics.net/lists/fio/msg07191.html . Without small fixed examples there's too much to explain.

There comes a point where there's nothing else to do other than read and understand the code of fio itself... Fio has main loop for submitting I/Os (see https://github.com/axboe/fio/blob/fio-3.8/backend.c#L1055 ):

static void do_io(struct thread_data *td, uint64_t *bytes_done)
    while ((td->o.read_iolog_file && !flist_empty(&td->io_log_list)) ||
        (!flist_empty(&td->trim_list)) || !io_issue_bytes_exceeded(td) ||
        td->o.time_based) {
        } else {
            ret = io_u_submit(td, io_u);

            if (should_check_rate(td))
                td->rate_next_io_time[ddir] = usec_for_io(td, ddir);

            if (io_queue_event(td, io_u, &ret, ddir, &bytes_issued, 0, &comp_time))

             * See if we need to complete some commands. Note that
             * we can get BUSY even without IO queued, if the
             * system is resource starved.
            full = queue_full(td) ||
                (ret == FIO_Q_BUSY && td->cur_depth);
            if (full || io_in_polling(td))
                ret = wait_for_completions(td, &comp_time);

The ioengine's queuing routine is invoked by a call chain from io_u_submit(). Assuming the ioengine is asynchronous, it may choose to just "queue" the I/Os within fio and then at a later point submit the whole lot down in one go (usually as a result of its getevents() function being called from a wait_for_completions() call chain). However we'll leave tracing through fio's code as an exercise for the reader.

if I want to write a benchmark program like fio, how would I control the IO queue depth?

You would need to mimic one of fio's (asynchronous) ioengines and have an event loop that was capable of (asynchronously) submitting I/Os AND checking for their completion. Once you had such a thing the idea that you only submit up to a particular depth would be easy - if at any point you have outstanding uncompleted I/O that matches (or exceeds if you aren't checking one by one) the chosen depth then you need to wait for something to be completed before you submit more.

You may find aio-stress.c in the Linux Test Project easier to understand/modify than fio if you're making a toy benchmark.

  • @ahmadi if you like the answer I think you can upvote on it and/or accept it to show your approval – Anon Aug 9 '18 at 9:19
  • because my reputation is not more than 15 it is not shown. – ahmadi Aug 9 '18 at 9:27
  • 1
    Ah. According to meta.stackexchange.com/questions/5234/… you can upvote until you have more than 15 rep but you should be still able to accept... – Anon Aug 9 '18 at 9:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.