3

On a server hosting a wide range of websites, I often see IO becoming a bottleneck without being able to identify the processes responsible of IO operations with tools such as iotop, iostat or sar.

I suspect that those processes are performing a lot of IO on metadata (reading and/or writing attributes, creating or removing a lot of empty files, etc). Unfortunately, it seems that those operations are not accounted "per processes", nor are IO performed using memory-mapped files (mmap).

My question is: Is there a way to monitor and/or account (for instance, using cgroups and blkio) IO per process or (maybe even better) per file, including io on metadata and memory-mapped files?

Currently, I'm trying to account "which requests hit the disk" using systemtap, probing handle_mm_fault() (mm/memory.c in the kernel) for major page faults, but I haven't been able to verify if manipulation of filesystem metadata generates page faults handled by this function.

Thank you for your insights!

Improve this question
3
  • I guess you have already checked but what is the swapping usage like in vmstat? Also, do you have a DBMS running and what IO scheduler do you?
    – UnX
    Commented Apr 1, 2014 at 0:07
  • There is no DBMS, just a webserver, but it is worth noting that the server is also an nfs server that can be highly solicited.
    – Martin Richard
    Commented Apr 1, 2014 at 8:12
  • Oh, and I forgot, there is no swap partition on the server.
    – Martin Richard
    Commented Apr 1, 2014 at 10:49

2 Answers 2

Reset to default
1

I came up with a systemtap script which is close to what I wanted to do, but it does not track writes.

The code is on a gist: https://gist.github.com/Martiusweb/10633360

Improve this answer
1
  • That's pretty good. Add some system() calls to rrdtool and you would have a pretty mean metric and graphical representation too.
    – ppetraki
    Commented Jun 17, 2014 at 15:42
0

Following my comment I have just remembered this great command : iotop 

iotop  watches I/O usage information output by the Linux kernel (requires
2.6.20 or later) and displays a table of current I/O usage by processes or 
threads on the system.

However it is usually not installed so since your server seems to be in production, you may want to have a look here if you can t install it.

Also lsof is a great command to check files opened by process.

Improve this answer
2
  • If you still can't find why and you are using a DBMS, read this :engineering.linkedin.com/performance/…
    – UnX
    Commented Apr 1, 2014 at 0:24
  • Thank you for your answer. I already use iotop and it seems that it doesn't give me enough information (Global read/write troughput is greater than the sum of what is accounted per process).
    – Martin Richard
    Commented Apr 1, 2014 at 8:10

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .