12
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          11.50    0.02    5.38    0.07    0.00   83.04

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sdc               0.01    89.92    0.26   41.59     3.36   457.19    22.01     0.23    5.60   0.09   0.38
sdb               0.10    15.59    0.40   14.55     8.96   120.57    17.33     0.04    2.91   0.07   0.11
sda               0.13    45.37    0.96    8.09    20.06   213.56    51.63     0.02    2.64   0.16   0.14
sde               0.01    31.83    0.09   11.34     0.94   103.56    18.29     0.04    3.52   0.14   0.16
sdd               0.01    48.01    0.13   19.81     1.58   202.16    20.44     0.11    5.62   0.13   0.25

Is there a way to know what files are being written? 457 kB/s

Also this other linux machine have this same problem.

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          20.50    0.00   46.48   20.74    0.00   12.28

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.17    11.61    0.99    3.51    36.65    59.43    42.70     0.10   23.20   3.84   1.73
sdb               0.55   224.18   24.30   97.45   246.48  1287.12    25.19     3.96   32.53   7.88  95.91
sdd               0.53   226.75   25.56   90.96   283.50  1271.69    26.69     3.43   29.44   8.22  95.75
sdc               0.00     1.76    0.28    0.06     4.73     7.26    70.41     0.00   12.00   2.12   0.07
dm-0              0.00     0.00    1.11   14.77    36.41    58.92    12.01     1.00   62.86   1.09   1.74
dm-1              0.00     0.00    0.04    0.12     0.17     0.49     8.00     0.00   21.79   2.47   0.04
dm-2              0.00     0.00    0.01    0.00     0.05     0.01     8.50     0.00    7.90   2.08   0.00

1200 write request per second for a server that host nothing

7

Well, you could try the following commands which worked for me in RHEL6:

1) Whatever device you see in "iostat" output performing more I/O, use it with fuser command as follows:

fuser -uvm device

2) You will get a list of processes with the user name causing more I/O. Select those PIDS and use it in the lsof command as follows:

lsof -p PID | more

3) You will get a list of files/directories along with the user performing maximum I/O.

  • root@host [~]# fuser -uv /home4 root@host [~]# fuser -uvm /home4 USER PID ACCESS COMMAND /home4: newgames 18297 ..c.. (newgames)php cuntschi 18407 ..c.. (cuntschi)php newgames 18457 ..c.. (newgames)php – user4951 Jan 6 '13 at 8:29
5

It seems there is no tools to find out I/O throughput per file other than within process using the file. But there are ways to find out process I/O throughput.

iotop - It is a top/iftop like utility that show process I/O throughput.

After pin pointing which process is having heavy I/O, use following to find out what file is being used

lsof -c <process name>

That should narrow down the scope and help pin point the source.

  • how to get the process? lsof also produce nothing. Something is writing so many things and I don't know what – user4951 Jan 6 '13 at 4:48
  • Sorry, I put the wrong io utility. It should be iotop, not iostat. I corrected my answer. – John Siu Jan 6 '13 at 4:50
  • (1) Can you also update what distro(redhat/centos/ubuntu ...) you are running? (2) Is this a fresh install? (3) Any file sharing like samba running? – John Siu Jan 6 '13 at 4:57
2

Use systemtap to monitor each write/pwrite syscall. You can do some accounting and you will see which file/fd gets the most bytes.

1

You can use inotifywait from inotify-tools to find out exactly which file is being written to. This won't tell you how much data is being written, but it will at least tell you what files are being written to as it happens.

As an example, this command will print the file name as soon as any are created, modified, or deleted in /tmp:

$ sudo inotifywait -e modify -e attrib -e move -e create -e delete -m -r /tmp
Setting up watches.  Beware: since -r was given, this may take a while!
Watches established.
/tmp/ CREATE test
/tmp/ MODIFY test

Unfortunately you will need to guess which directory contains the files being written to. This fails if you try to use it on the root directory, though apparently that can be overridden:

$ sudo inotifywait -e modify -e attrib -e move -e create -e delete -m -r /
Setting up watches.  Beware: since -r was given, this may take a while!
Failed to watch /; upper limit on inotify watches reached!
Please increase the amount of inotify watches allowed per user via `/proc/sys/fs/inotify/max_user_watches'.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.