0

I am conducting a performance experiment for a Linux program that takes input data, processes it, and writes data to a block device

Even though I know how much data the input is (for example, 1GB), I do not know exactly how much data is written to the block device since the program will process it (including deleting some intermediate data, write extra data).So this is not good to calculate the entire throughput for the block device.

I once used blktrace, but it can easily get tons of data, which I am not sure is able to work for the past 1hr.

I know FIO is able to benchmark IO device, but I specifically want to test the block IO device behavior under this program. So basically I am actually testing the combination of block IO device and the program instead of just the block IO device.

I thought about the du -sh approach as well, but du -sh is unable to catch the deleted data amount. Besides, can du -sh catch the data written to the physical device? I mean, not to the file system.

Is there any other tool to use?

Improve this question
4
  • The sysstat package contains some utilities you might look in to like; iostat and sar. There is also a tool called iotop that you may find useful. Several answers in a similar vein that may help you here: unix.stackexchange.com/questions/55212/…
    – Csnap
    May 20 at 18:52
  • Thanks. It seems all these tools report bytes per second, is there any tool that print just total bytes in a time window that I specified ? Or probably I missed some command line flags in them?
    – Dachuan Huang
    May 20 at 23:15
  • You could always just run smartctl against the device and parse the output.
    – Bib
    May 21 at 10:05
  • The question is not clear. Do you want only to know how much this specific program wrote to the disk? You also said that the program might "delete some intermediate data, write extra data". Why does it matter? If the program wrote intermediate data and deleted it, you don't want it to be included in the calculation? And what does "extra data" even mean? What types of "data" are there?
    – aviro
    May 22 at 15:51

1 Answer 1

Reset to default
1

If you need the total bytes read/write per hour, iostat can provide you that.

$ iostat sda -tdyh 10
Linux 4.12.14-122.136-default (hostname)       05/22/2023      _x86_64_        (18 CPU)

05/22/2023 12:56:22 PM
      tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn Device
     2.30         0.0k        14.8k       0.0k     148.0k sda

05/22/2023 12:56:32 PM
      tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn Device
   412.00         0.0k       102.5M       0.0k       1.0G sda

05/22/2023 12:56:42 PM
      tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn Device
   420.80       102.4M       104.8k       1.0G       1.0M sda

kB_read and kB_wrtn show the total number of blocks read or written respectively. First instance, it shows that:

  • Between 12:56:22 and 12:56:32 1GB were written to the disk.
  • Between 12:56:32 and 12:56:42 1GB were read from the disk.

(I've used the dd command to write/read 1G file to the disk to create those numbers and ensure the report is correct)

The flags I've used:

  • -t - Print the time for each report displayed.
  • -d - Display only device report (without it, iostat will also display CPU utilization)
  • -y - Omit first report with statistics since system boot. Will only report the utilization between the intervals.
  • -h - Make the Device Utilization Report easier to read by a human (MB / GB if needed, and not just kB)
  • 10 - is the interval (in seconds), in this case every 10 seconds. If you want to check every hour, use 3600.
Improve this answer
1
  • Note that the data written.red will include meta-data and (on most filesystems) journalling.
    – symcbean
    May 22 at 13:58

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .