Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

On a server hosting a wide range of websites, I often see IO becoming a bottleneck without being able to identify the processes responsible of IO operations with tools such as iotop, iostat or sar.

I suspect that those processes are performing a lot of IO on metadata (reading and/or writing attributes, creating or removing a lot of empty files, etc). Unfortunately, it seems that those operations are not accounted "per processes", nor are IO performed using memory-mapped files (mmap).

My question is: Is there a way to monitor and/or account (for instance, using cgroups and blkio) IO per process or (maybe even better) per file, including io on metadata and memory-mapped files?

Currently, I'm trying to account "which requests hit the disk" using systemtap, probing handle_mm_fault() (mm/memory.c in the kernel) for major page faults, but I haven't been able to verify if manipulation of filesystem metadata generates page faults handled by this function.

Thank you for your insights!

share|improve this question
    
I guess you have already checked but what is the swapping usage like in vmstat? Also, do you have a DBMS running and what IO scheduler do you? –  rMistero Apr 1 at 0:07
    
There is no DBMS, just a webserver, but it is worth noting that the server is also an nfs server that can be highly solicited. –  Martin Richard Apr 1 at 8:12
    
Oh, and I forgot, there is no swap partition on the server. –  Martin Richard Apr 1 at 10:49

2 Answers 2

up vote 1 down vote accepted

I came up with a systemtap script which is close to what I wanted to do, but it does not track writes.

The code is on a gist: https://gist.github.com/Martiusweb/10633360

share|improve this answer
    
That's pretty good. Add some system() calls to rrdtool and you would have a pretty mean metric and graphical representation too. –  ppetraki Jun 17 at 15:42

Following my comment I have just remembered this great command : iotop

iotop  watches I/O usage information output by the Linux kernel (requires
2.6.20 or later) and displays a table of current I/O usage by processes or 
threads on the system.

However it is usually not installed so since your server seems to be in production, you may want to have a look here if you can t install it.

Also lsof is a great command to check files opened by process.

share|improve this answer
    
If you still can't find why and you are using a DBMS, read this :engineering.linkedin.com/performance/… –  rMistero Apr 1 at 0:24
    
Thank you for your answer. I already use iotop and it seems that it doesn't give me enough information (Global read/write troughput is greater than the sum of what is accounted per process). –  Martin Richard Apr 1 at 8:10

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.