9

is there any way to see the disk I/O per pid/process ? Monitorix shows me the following and correct me if I´m wrong, but it looks like that the harddrive is slowing down the whole system: enter image description here

UPDATE #1

Atop seems to give a great overview above everything.

  • Might I ask you what the wait I marked red means?
  • Is that the wait from the disk I/O?

enter image description here

4
  • 2
    You have either atop, iotop or pidstat
    – sebelk
    Mar 4, 2014 at 22:36
  • 1
    @sebelk Hey, I updated the question, it would be nice if you can take a look. :) Mar 4, 2014 at 23:00
  • 1
    it was not realy possible to select a best answer so I decided to take the one which helped me the most and upvoted the other ones. Mar 5, 2014 at 22:50
  • @user2693017 - that's completely fine, as the OP it's your prerogative which A best suited your needs.
    – slm
    Mar 6, 2014 at 1:35

3 Answers 3

8

I believe there are a lot tools. One of them is pidstat:

pidstat -d 5

Shows I/O of processes every 5 seconds.

Example

$ pidstat -d 5
Linux 3.12.11-201.fc19.x86_64 (greeneggs.bubba.net)     03/04/2014  _x86_64_    (4 CPU)

06:19:05 PM   UID       PID   kB_rd/s   kB_wr/s kB_ccwr/s  Command
06:19:10 PM  1000     29486      0.00    317.93      0.00  chrome

06:19:10 PM   UID       PID   kB_rd/s   kB_wr/s kB_ccwr/s  Command
06:19:15 PM  1000     29486      0.00    368.00      0.00  chrome

06:19:15 PM   UID       PID   kB_rd/s   kB_wr/s kB_ccwr/s  Command
06:19:20 PM  1000     29486      0.00    591.62    836.73  chrome

06:19:20 PM   UID       PID   kB_rd/s   kB_wr/s kB_ccwr/s  Command
06:19:25 PM  1000      2410      0.00      0.80      0.00  mono
06:19:25 PM  1000     29486      0.00    262.40      0.00  chrome
^C

Average:      UID       PID   kB_rd/s   kB_wr/s kB_ccwr/s  Command
Average:     1000      2410      0.00      0.20      0.00  mono
Average:     1000     29486      0.00    385.02    209.29  chrome

From the point where you see the Ctrl+C (aka. ^C) is the totals for the duration of the run above.

2
  • nice, is it possible to let it run for 1 hour and get the overall results? Mar 5, 2014 at 1:06
  • perhaps, I didn't tried it yet. You can also define the PID's you want to inspect.
    – user55518
    Mar 5, 2014 at 10:58
3

It's difficult for me to look well at this pictures, but:

Take a look at man atop:

I wonder if you finally are asking for another thing, in the CPU section you have:

Every  line contains the percentage of cpu time spent in kernel mode by all
active processes (`sys'), the percentage of cpu time consumed in user mode
(`user') for all active processes (including processes running with a nice
value larger than zero), the  percentage  of  cpu  time spent  for  interrupt
handling  (`irq')  including  softirq, the percentage of unused cpu time while
no processes were waiting for disk-I/O (`idle'), and the percentage of unused
cpu time while at least one process was waiting for disk-I/O (`wait').

In case of per-cpu occupation, the last column shows the cpu number and the
wait percentage (`w') for that cpu.  The number of lines showing the per-cpu
occupation can be limited.

Anyway you may read the disk-specific statistics:

d    Show disk-related output.

            When "storage accounting" is active in  the  kernel,  the  
            following  fields  are shown: process-id, amount of data read 
            from disk, amount of data written to disk, amount of data that 
            was written but has been withdrawn again (WCANCL), disk 
            occupation percentage and process name.

As well as these options.

D

    D   Sort  the  current  list  in the order of disk accesses issued.  
        The one-but-last column changes to ``DSK''.
        ...

RDDSK

RDDSK   When the kernel maintains standard io statistics (>= 2.6.20):

        The  read  data  transfer issued physically on disk (so reading from 
        the disk cache is not accounted for).
        ...

WRDSK

WRDSK   When the kernel maintains standard io statistics (>= 2.6.20):

        The  write  data  transfer  issued physically on disk (so writing to 
        the disk cache is not accounted for).  This counter is maintained 
        for the  application process  that writes its data to the cache 
        (assuming that this data is physically transferred to disk later 
        on). Notice that disk I/O needed for swapping is not taken into 
        account.
        ....

LVM/MDD/DSK

LVM/MDD/DSK
        Logical volume/multiple device/disk utilization. 

        Per active unit one line is produced, sorted on unit activity.
        Such  line shows the name (e.g. VolGroup00-lvtmp for a logical
        volume or sda for a hard disk), the busy percentage i.e. the
        portion of time that the unit was busy handling requests
        (`busy'),  the  number  of  read  requests  issued  (`read'), the
        number of write requests issued (`write'), the number of KiBytes
        per read (`KiB/r'),  the  number of  KiBytes  per write
        (`KiB/w'), the number of MiBytes per second throughput for reads
        (`MBr/s'),  the  number  of  MiBytes  per second  throughput  for
        writes (`MBw/s'), the average queue depth (`avq') and the average
        number of milliseconds needed by a request (`avio') for seek,
        latency and data transfer.

        If the screen-width does not allow all of these counters, only a
        relevant subset is shown.

        The  number of lines showing the units can be limited per class
        (LVM, MDD or DSK) with the 'l' key or statically (see separate
        man-page of atoprc).  By specifying the  value  0  for  a
        particular class, no lines will be shown any more for that class.
1
3

Take a look at this U&L Q&A titled: Amazon EC2 micro instance large number of IO requests, where I provide details around a tool called fatrace. I also cover it in this Q&A titled: Determining Specific File Responsible for High I/O.

fatrace

This is a new addition to the Linux Kernel and a welcomed one, so it's only in newer distros such as Ubuntu 12.10. My Fedora 14 system was lacking it 8-).

It provides the same access that you can get through inotify without having to target a particular directory and/or files.

$ sudo fatrace
pickup(4910): O /var/spool/postfix/maildrop
pickup(4910): C /var/spool/postfix/maildrop
sshd(4927): CO /etc/group
sshd(4927): CO /etc/passwd
sshd(4927): RCO /var/log/lastlog
sshd(4927): CWO /var/log/wtmp
sshd(4927): CWO /var/log/lastlog
sshd(6808): RO /bin/dash
sshd(6808): RO /lib/x86_64-linux-gnu/ld-2.15.so
sh(6808): R /lib/x86_64-linux-gnu/ld-2.15.so
sh(6808): O /etc/ld.so.cache
sh(6808): O /lib/x86_64-linux-gnu/libc-2.15.so

The above shows you the process ID that's doing the file accessing and which file it's accessing, but it doesn't give you any overall bandwidth usage, so each access is indistinguishable to any other access.

NOTE: fatrace can take a -p PID argument so you can direct it to watch a single PID if you want instead.

4
  • Really nice tool, thanks I didn't know about of it!
    – sebelk
    Mar 5, 2014 at 12:42
  • you have a related and nice tool that is ftop
    – sebelk
    Mar 5, 2014 at 12:50
  • @sebelk - yup thanks, have that installed already too.
    – slm
    Mar 5, 2014 at 13:20
  • I like this tool. It shows which task accesses which files on disk. These files can however also be mmaped, so only buffer access happens and no IO access.
    – user55518
    Mar 6, 2014 at 0:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.