What is the impact of different Linux I/O APIs on performance benchmarking of NVMe SSDs? (libaio vs SPDK vs io_uring)

Question

I am benchmarking NVMe SSDs on my Linux server, with the aim to achieve the IOPS, BW and Latency values as mentioned in the product specifications.

I'm using FIO as workload generator, and used libaio as the I/O Engine initially, but the more I dived deep into the storage domain, I realize that it is an outdated API which is broken-beyond-repair, due to its limitations of providing async operations only for unbuffered access, and huge overhead of syscalls.

This paper: "Understanding modern storage APIs: a systematic study of libaio, SPDK, and io_uring", helped me get an overall understanding of backend operations of completing an I/O command, and comparison between the three by sweeping on number of Devices, Core Count & (software) Queue Depth. However, a few parameters are left unaddressed due to it not being in the scope of application development perspective, such as impact of number of jobs (which act as different workers/processes), threads, different block sizes, and test type (seq/random R/W) which heavily impact the numbers.

My question is: Is every library capable of saturating the SSD (i.e. delivering the numbers mentioned in spec)? If yes, then is it okay to benchmark with any of the given libraries? With benchmarking, here I mean making a profile of ways to achieve max throughput and min latency in different configurations (like single-core/multi-core, single-thread/multi-thread, combination of R/W etc). If not, then does it mean that outdated tech like libaio can never match the modern device speeds, and it should no longer be used (since it is unfair to define device speeds using something that bottlenecks at OS level)?

I believe benchmarking should only be done by the API which is also used in the application for which the device is being benchmarked. But this feels counter-intuitive, since benchmarking a hardware device should not be affected by unrelated OS operations, which is not even a property of application. I'd also like to know ways of finding latency overhead in each step of the operation as compared to a plain sync I/O.

I feel the things around here are under-documented and would love to gain a deeper understanding and thoughts of experienced people in this domain. Thanks!

Answer 1

My question is: Is every library capable of saturating the SSD (i.e. delivering the numbers mentioned in spec)?

io_uring has less overhead, so it can be faster. But if your CPU is fast enough, libaio can also saturate a SSD.

My tests:
-ioengine=libaio -numjobs=1:

   iops        : min=286708, max=386670, avg=381760.92, stdev=13237.78, samples=59

-ioengine=io_uring -numjobs=1:

   iops        : min=399008, max=414024, avg=409571.12, stdev=2658.33, samples=59

(+7%)

-ioengine=libaio -numjobs=2:

   iops        : min=779822, max=814588, avg=805774.92, stdev=2274.38, samples=118

-ioengine=io_uring -numjobs=2:

   iops        : min=438596, max=853962, avg=810650.24, stdev=38096.63, samples=118

(less than 1% difference)

-ioengine=libaio -numjobs=4:

   iops        : min=843282, max=1160188, avg=1131725.17, stdev=12340.14, samples=236

-ioengine=io_uring -numjobs=4:

   iops        : min=1031955, max=1163986, avg=1140789.00, stdev=5196.52, samples=236

(less than 1% difference again)

Increasing numjobs further doesn't change anything, it seems like the SSD is saturated.

fio full command line:

fio -ioengine=XXX -direct=1 -name=test -bs=4k -iodepth=32 -rw=randread -runtime=30 -filename=/dev/nvmeYYY -numjobs=ZZZ -group_reporting

The SSD is Samsung pm9a3