On a server hosting a wide range of websites, I often see IO becoming a bottleneck without being able to identify the processes responsible of IO operations with tools such as iotop, iostat or sar.

I suspect that those processes are performing a lot of IO on metadata (reading and/or writing attributes, creating or removing a lot of empty files, etc). Unfortunately, it seems that those operations are not accounted "per processes", nor are IO performed using memory-mapped files (mmap).

My question is: Is there a way to monitor and/or account (for instance, using cgroups and blkio) IO per process or (maybe even better) per file, including io on metadata and memory-mapped files?

Currently, I'm trying to account "which requests hit the disk" using systemtap, probing handle_mm_fault() (mm/memory.c in the kernel) for major page faults, but I haven't been able to verify if manipulation of filesystem metadata generates page faults handled by this function.

Thank you for your insights!

  • I guess you have already checked but what is the swapping usage like in vmstat? Also, do you have a DBMS running and what IO scheduler do you? – UnX Apr 1 '14 at 0:07
  • There is no DBMS, just a webserver, but it is worth noting that the server is also an nfs server that can be highly solicited. – Martin Richard Apr 1 '14 at 8:12
  • Oh, and I forgot, there is no swap partition on the server. – Martin Richard Apr 1 '14 at 10:49

I came up with a systemtap script which is close to what I wanted to do, but it does not track writes.

The code is on a gist: https://gist.github.com/Martiusweb/10633360

  • That's pretty good. Add some system() calls to rrdtool and you would have a pretty mean metric and graphical representation too. – ppetraki Jun 17 '14 at 15:42

Following my comment I have just remembered this great command : iotop

iotop  watches I/O usage information output by the Linux kernel (requires
2.6.20 or later) and displays a table of current I/O usage by processes or 
threads on the system.

However it is usually not installed so since your server seems to be in production, you may want to have a look here if you can t install it.

Also lsof is a great command to check files opened by process.

  • If you still can't find why and you are using a DBMS, read this :engineering.linkedin.com/performance/… – UnX Apr 1 '14 at 0:24
  • Thank you for your answer. I already use iotop and it seems that it doesn't give me enough information (Global read/write troughput is greater than the sum of what is accounted per process). – Martin Richard Apr 1 '14 at 8:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.