On a server hosting a wide range of websites, I often see IO becoming a bottleneck without being able to identify the processes responsible of IO operations with tools such as iotop, iostat or sar.
I suspect that those processes are performing a lot of IO on metadata (reading and/or writing attributes, creating or removing a lot of empty files, etc). Unfortunately, it seems that those operations are not accounted "per processes", nor are IO performed using memory-mapped files (mmap).
My question is: Is there a way to monitor and/or account (for instance, using cgroups and blkio) IO per process or (maybe even better) per file, including io on metadata and memory-mapped files?
Currently, I'm trying to account "which requests hit the disk" using systemtap, probing handle_mm_fault() (mm/memory.c in the kernel) for major page faults, but I haven't been able to verify if manipulation of filesystem metadata generates page faults handled by this function.
Thank you for your insights!