How to monitor size of the storage iops on a modern Linux ?

I'm able to monitor quantity of the storage iops using commands like iostat. However I would like to know the size of the iops which are generated by the workload.

Something like, in last X seconds there were: 5 iops 4K, 10 iops 8K, 20 iops 16K ...

I'm interested in monitoring at block device level (e.g.: /dev/sda) however at any other level would be interesting, as well.

  • What OS? Solaris (dtrace) or Linux (???) ... – thrig Jun 1 '17 at 17:02
  • yeah, need more info on the OS. Need more info on if its a RAID volume, ZFS volume, CEPH volume... What are we dealing with her? – AfroJoe Jun 1 '17 at 17:33
  • @thrig thank you, I have expanded. Thank you for pointing in right direction, histogram of write I/O sizes like using DTrace would be great but on linux :-) – TimB Jun 1 '17 at 17:57
  • @AfroJoe, thank you, I'm interested at any level, however block device would be preferable. – TimB Jun 1 '17 at 17:57
  • github.com/iovisor/bcc has a block I/O latency histogram thingy but you might need a not-ancient kernel to use BPF. – thrig Jun 1 '17 at 18:06

Low level

At the low level, you can get this information from /sys/block/X/stat, which is documented here.

Example content looks like:

   10465     3250   759844    27528     4400     2983   546848    66306        0     9754    93792

The fields we want are columns 1 & 3 for reads, and 5 & 7 for writes. The calculation method is the same for both, so I'll just discuss reads.

Column 1 is the number of reads (which you stated you're already able to monitor), and column 3 is the number of sectors read. A sector is 512 bytes. So with these to calculate the size of the reads, you just do $col3 * 512 / $col1.
The numbers in this file are the total since disk came online. So to calculate the average read size over a period of time, you just read the file twice, and calculate with: ($col3_t2 - $col3_t1) * 512 / ($col1_t2 - $col1_t1).

Note that the above method gives you the size of the OPs performed at the application level (description is not exact, but close enough). The kernel can merge multiple operations together so the number of operations performed on the disk is smaller. The calculation for this is: $col3 * 512 / ($col1 - $col2). And to get the value over a span of time: ($col3_t2 - $col3_t1) * 512 / (($col1_t2 - $col1_t1) - ($col2_t2 - $col2_t1))

High level

iostat can give you the value as well, but less accurately. It doesn't differentiate writes or reads, and it only gives you the value in sectors, not bytes (so again, multiply by 512). But the command is iostat -x -d, and the column to look at is avgrq-sz.

  • this provides average size of iops based on total size of iops and total count. I'm looking more for a "histogram" type data which will allow me to see what sizes of iops are generated by workload. – TimB Jun 5 '17 at 20:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.