I'm trying to figure out the completion latency of fio benchmark with NVMe SSD.

I made following fio script to test benchmark in fio.

I used following options.

rw=read, ioengine=sync, direct=1

So, I thought there's not much things to make completion times different.

However, the result wasn't what I thought.

enter image description here

The result is 1th : 11us to 99.99th : 111 us.

Synchronous read make no outstanding IOs, so all I/Os are processed sequentially,

And direct option can bypass the buffer in OS.

I thought most of latencies are same.

Any ideas for this result?


You are talking about microseconds here and 11 microseconds vs 111 microseconds. It isn't going to take much to perturb such a sensitive reading:

  • fio process goes off CPU because your server decided it was time to do a periodic cron job and doesn't get to process some I/O as quickly as others
  • Some of the I/O was in some sort of SSD cache but later I/O actually had to fetched from outside the cache
  • Some of the I/O read happened to have been written in a different order to which is read back (yes reading things back in the order they are written is better for an SSD)
  • Something else decided to I/O to the same device


You didn't include the full job (and I can see you set iodepth=16 which leaves me wondering what other things you set on your job) so you're asking a bit much from any answer. However I'll point out that 95% of your readings are within 3 microseconds of each other so you're not seeing many outliers across just under a million I/Os. How much determinism are you expecting out of a non-realtime system?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.