2

Is iodepth of fio and queue depth of storage both same? Then, how is it possible to control queue depth with an iodepth parameter from fio command? Will that be creating parallel jobs, but then again there is an option to run jobs in parallel also (will that not be a trivial or conflicting?)

I am struggling to understand how fio is controlling it's workloads (especially about this iodepth). Can someone please explain the iodepth parameter in detail.

UPDATE #1

My question was also asked in the Flexible I/O Tester's forum. This is the answer that I received there.

Hi,

On 28 July 2018 at 14:26, Jeevan Patnaik wrote: Hi,

Is Iodepth of fio and queue depth of storage both same? Then, how is it possible to control queue depth with an iodepth parameter from fio

fio iodepth and the depth I/O your OS achieves submitting I/O down to storage are linked but most certainly do not have to be the same and the relationship is highly dependent on your operating system/storage/fio ioengine used/fio parameters. Basically fio submits I/O a particular way to your operating system. Depending on how you submit your I/O to your operating system it can choose to submit it further down in a more optimal/different fashion (e.g. by batching requests together, breaking requests that are too big into smaller pieces, delaying I/O etc). Additionally and as stated in the HOWTO, iodepth only affects asynchronous ioengines (and note that text includes warnings about the need to use direct=1 on Linux).

command? Will that be creating parallel jobs, but then again there is an option to run jobs in parallel also (will that not be a trivial or conflicting?)

I'm going to give a brief summary but note I'm not trying to cover caching/readahead/plugging/block device layers (e.g. RAID/LVM) etc:

A synchronous fio I/O engine submits a single I/O to the OS, waits for it to be "acknowledged" as having been received and then sends another I/O etc.

If an fio I/O engine is able to submit I/O to the OS in a truly asynchronous fashion (see link above) then the key is that it does NOT have to wait for earlier I/O to be "acknowledged" before submitting new I/O. If the iodepth is only 1 it will have to behave in a fashion similar to a synchronous I/O engine. However, let's say a jobs specifies an iodepth of 32. In that case up to 32 I/Os to be outstanding before fio will choose to wait before submitting any more I/O (just what the watermarks are and how much is submitted at a time is controlled by the iodepth_batch_* options. This can be more efficient and achieve higher throughputs but often comes with a cost of higher latency.

fio will not create parallel fio jobs just because of iodepth BUT using parallel fio jobs is another way of increasing the amount of simultaneous I/O being submitted at any given time (by using different threads/processes) and using both on the same device will act in tandem (so if you have two fio jobs submitting asynchronous I/O at an iodepth of 16 each your OS could be be actually receiving 32 I/Os at any given time). There can be reasons for combining the two (e.g. you have multiple devices and they are so fast that one CPU can't keep up even when submitting I/O asynchronously).

I am struggling to understand how fio is controlling it's workloads (especially about this iodepth). Can someone please explain the iodepth parameter in detail.

I will note you've also asked this question over on stackexchange (What exactly is iodepth in fio?). You may want to link to https://www.spinics.net/lists/fio/msg07190.html from there to help others who may have a similar question...

closed as off-topic by Jeff Schaller, schily, Rui F Ribeiro, Jesse_b, Romeo Ninov Aug 4 '18 at 16:14

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "This question has been posted on multiple sites. Cross-posting is strongly discouraged; see the help center and community FAQ for more information." – Jeff Schaller, schily, Rui F Ribeiro, Jesse_b, Romeo Ninov

1

will that not be a trivial?

Assume direct IO, as required for iodepth= to work.

A sequential job with iodepth=2 will submit two sequential IO requests at a time.

A sequential job with numjobs=2 will have two threads, each submitting sequential IO.

These are different IO patterns. The latter will generate 2x the bandwidth across the IO bus, even if the physical IO reduces back to 1x due to device caches. (I suspect the two jobs would tend to remain in lockstep due to device caches, unless you used multiple files and a randomized file_service_type=). If the IOs are synchronous writes (sync=true), the physical IO would not be reduced at all, unless the device is doing an unusual amount of optimization (perhaps a de-duplicating SSD controller).

0

Per the Linux kernel docs:

.. option:: iodepth=int

Number of I/O units to keep in flight against the file. Note that increasing iodepth beyond 1 will not affect synchronous ioengines (except for small degrees when :option:verify_async is in use). Even async engines may impose OS restrictions causing the desired depth not to be achieved. This may happen on Linux when using libaio and not setting :option:direct\=1, since buffered I/O is not async on that OS. Keep an eye on the I/O depth distribution in the fio output to verify that the achieved depth is as expected. Default: 1.

This tutorial titled: Fio Output Explained had this example:

Fio has an iodepth setting that controls how many IOs it issues to the OS at any given time. This is entirely application-side, meaning it is not the same thing as the device's IO queue. In this case, iodepth was set to 1 so the IO depth was always 1 100% of the time.

    submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
    complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

submit and complete represent the number of submitted IOs at a time by fio and the number completed at a time. In the case of the thrashing test used to generate this output, the iodepth is at the default value of 1, so 100% of IOs were submitted 1 at a time placing the results in the 1-4 bucket. Basically these only matter if iodepth is greater than 1.

  • (I think specifically the intent was that io_submit() returns -EAGAIN when the queue is full, but that was only implemented recently, and before then it would end up blocking. lwn.net/Articles/724198 ) – sourcejedi Jul 28 '18 at 21:35
  • @sourcejedi numjobs option also does the samething right? It allows jobs to be submitted in parallel. If these parallel jobs can also create the same queue depth at the storage level, why iodepth again? – GP92 Jul 29 '18 at 6:51

Not the answer you're looking for? Browse other questions tagged or ask your own question.