1

I am benchmarking NVMe SSDs on my Linux server, with the aim to achieve the IOPS, BW and Latency values as mentioned in the product specifications.

I'm using FIO as workload generator, and used libaio as the I/O Engine initially, but the more I dived deep into the storage domain, I realize that it is an outdated API which is broken-beyond-repair, due to its limitations of providing async operations only for unbuffered access, and huge overhead of syscalls.

This paper: "Understanding modern storage APIs: a systematic study of libaio, SPDK, and io_uring", helped me get an overall understanding of backend operations of completing an I/O command, and comparison between the three by sweeping on number of Devices, Core Count & (software) Queue Depth. However, a few parameters are left unaddressed due to it not being in the scope of application development perspective, such as impact of number of jobs (which act as different workers/processes), threads, different block sizes, and test type (seq/random R/W) which heavily impact the numbers.

My question is: Is every library capable of saturating the SSD (i.e. delivering the numbers mentioned in spec)? If yes, then is it okay to benchmark with any of the given libraries? With benchmarking, here I mean making a profile of ways to achieve max throughput and min latency in different configurations (like single-core/multi-core, single-thread/multi-thread, combination of R/W etc). If not, then does it mean that outdated tech like libaio can never match the modern device speeds, and it should no longer be used (since it is unfair to define device speeds using something that bottlenecks at OS level)?

I believe benchmarking should only be done by the API which is also used in the application for which the device is being benchmarked. But this feels counter-intuitive, since benchmarking a hardware device should not be affected by unrelated OS operations, which is not even a property of application. I'd also like to know ways of finding latency overhead in each step of the operation as compared to a plain sync I/O.

I feel the things around here are under-documented and would love to gain a deeper understanding and thoughts of experienced people in this domain. Thanks!

3
  • this feels counter-intuitive, since benchmarking a hardware device should not be affected by unrelated OS operations A benchmark is a measure of performance in a given use case. The use case includes the application requesting IO, and the operating system executing that. So, this is not surprising at all! Just like you can't say how fast a car can complete a race to deliver parcels without considering both the driver and the goods it's transporting. Dec 17, 2022 at 18:24
  • @MarcusMüller Ah right! I understand what you're saying now. So why isn't this much more discussed phenomenon then?? We don't discuss which library is being used to benchmark certain device or make a profile of configurations and it is rather discussed in an unrelated way. Shouldn't it be very much affected by the API types? Is this some gap that needs to be filled (benchmarking standards across different APIs defined explicitly) or am I missing something? Dec 19, 2022 at 16:48
  • Hi guys! I have started a bounty and I am looking forward to listening your feedbacks on how I can make this question clearer. I am new to storage technologies and apologize for vagueness in my question. Dec 20, 2022 at 14:02

1 Answer 1

3

My question is: Is every library capable of saturating the SSD (i.e. delivering the numbers mentioned in spec)?

io_uring has less overhead, so it can be faster. But if your CPU is fast enough, libaio can also saturate a SSD.

My tests:
-ioengine=libaio -numjobs=1:

   iops        : min=286708, max=386670, avg=381760.92, stdev=13237.78, samples=59

-ioengine=io_uring -numjobs=1:

   iops        : min=399008, max=414024, avg=409571.12, stdev=2658.33, samples=59

(+7%)

-ioengine=libaio -numjobs=2:

   iops        : min=779822, max=814588, avg=805774.92, stdev=2274.38, samples=118

-ioengine=io_uring -numjobs=2:

   iops        : min=438596, max=853962, avg=810650.24, stdev=38096.63, samples=118

(less than 1% difference)

-ioengine=libaio -numjobs=4:

   iops        : min=843282, max=1160188, avg=1131725.17, stdev=12340.14, samples=236

-ioengine=io_uring -numjobs=4:

   iops        : min=1031955, max=1163986, avg=1140789.00, stdev=5196.52, samples=236

(less than 1% difference again)

Increasing numjobs further doesn't change anything, it seems like the SSD is saturated.

fio full command line:

fio -ioengine=XXX -direct=1 -name=test -bs=4k -iodepth=32 -rw=randread -runtime=30 -filename=/dev/nvmeYYY -numjobs=ZZZ -group_reporting

The SSD is Samsung pm9a3

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .