IOStat report large ocassional writes but find reports no big file

Question

root@host [/home4]# find . -type f -size +2000k -exec ls -lh {} \; | awk '{ print $9 ": " $5 }'
./cuntschi/public_html/hello/cache/curl/http%3A%2F%2Ftransport01.files.wordpress.com%2F2009%2F12%2Ftransportenlogistiek.doc: 2.0M
./newgames/public_html/hello/cache/curl/http%3A%2F%2Fwww.wincouncil.org.sg%2Fimages%2Fnewsroom%2Fnewsletter%2Fnovember-2008.pdf: 2.2M

yet iostat -x 1 shows:

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sdc               0.00  1878.00    8.00  988.00   152.00 22944.00    23.19     7.24    7.27   0.08   7.90
sdb               0.00     0.00   17.00    0.00   216.00     0.00    12.71     0.01    0.35   0.18   0.30
sda               0.00     0.00    2.00   25.00    40.00   200.00     8.89     0.02    0.85   0.15   0.40
sde               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdd               0.00     0.00   10.00    0.00   176.00     0.00    17.60     0.01    1.40   1.20   1.20

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          57.14    0.00   16.79    0.25    0.00   25.81

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sdc               0.00     0.00   20.00    0.00   528.00     0.00    26.40     0.00    0.20   0.15   0.30
sdb               0.00     0.00    8.00    0.00   152.00     0.00    19.00     0.00    0.50   0.50   0.40
sda               0.00     0.00    2.00    0.00    40.00     0.00    20.00     0.00    1.00   1.00   0.20
sde               0.00   191.00    1.00  231.00    16.00  3376.00    14.62     0.70    3.03   0.06   1.40
sdd               0.00     0.00   14.00    0.00   208.00     0.00    14.86     0.01    0.93   0.50   0.70

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          68.20    0.00   22.07    0.25    0.00    9.48

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sdc               0.00     0.00   31.00    0.00   736.00     0.00    23.74     0.01    0.19   0.16   0.50
sdb               0.00   121.00    7.00   46.00   136.00  1336.00    27.77     0.08    1.49   0.15   0.80
sda               0.00     0.00    0.00   34.00     0.00   272.00     8.00     0.03    1.00   0.06   0.20
sde               0.00     0.00    5.00    0.00    96.00     0.00    19.20     0.00    0.80   0.80   0.40
sdd               0.00   583.00    4.00  452.00    72.00  8280.00    18.32     3.96    8.70   0.11   5.20

So basically we got ocassional 22MB files (how else it's 22944/s but there are few files no file bigger than 2 MB

#
# /etc/fstab
# Created by anaconda on Thu Jan  3 07:08:05 2013
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
UUID=44882d5f-91fd-4409-9add-4393e7bd9f99       /       ext4    usrjquota=quota.user,jqfmt=vfsv0        1       1
UUID=ef4193db-62ef-45f0-bb56-98ab35729a7c /boot                   ext3    defaults        1 2
UUID=26eeb365-46c8-4da2-b326-908dedbf91e1 swap                    swap    defaults        0 0
tmpfs                   /dev/shm                tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
/usr/tmpDSK             /tmp                    ext3    defaults,noauto        0 0
/dev/sdb1               /home2                  auto    auto,defaults,relatime         0 0
/dev/sdc1               /home3                  auto    auto,defaults,relatime         0 0
/dev/sdd1               /home4                  auto    auto,defaults,relatime         0 0
/dev/sde1               /home5                  auto    auto,defaults,relatime         0 0

Answer 1

It is not a straight forward answer. I will try to list all I can base on all your questions.

1. Apache log every single access to log files.
2. WordPress plugin like jetpack (and many others) come with logging capability and is writing to database.
3. Though your mount option is relatime, every time file is written to, atime is also updated.
4. Journaled quota is used, every time file is modified, quota info is updated.

So it is a combination of write request from different part of your system.

PS1: (I may be wrong on this point) In iostat, wsec is write sector, which may not reflect the actual amount data being change. Especially if it is a lot of small changes to different files. Writing to a sector may not increase file size as it maybe changing sometime in the middle of the file, like database. Updating atime of file also does not change file size.

PS2: One possible way to reduce the number of write request is change relatime to noatime in your /etc/fstab. So system is no longer updating access time of each file being read/write.

Use noatime

Add noatime to /, replace relatime with noatime for all home mount point.

UUID=44882d5f-91fd-4409-9add-4393e7bd9f99       /       ext4    usrjquota=quota.user,jqfmt=vfsv0,noatime        1       1
UUID=ef4193db-62ef-45f0-bb56-98ab35729a7c /boot                   ext3    defaults        1 2
UUID=26eeb365-46c8-4da2-b326-908dedbf91e1 swap                    swap    defaults        0 0
tmpfs                   /dev/shm                tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
/usr/tmpDSK             /tmp                    ext3    defaults,noauto        0 0
/dev/sdb1               /home2                  auto auto,defaults,noatime         0 0
/dev/sdc1               /home3                  auto        auto,defaults,noatime         0 0
/dev/sdd1               /home4                  auto   auto,defaults,noatime         0 0
/dev/sde1               /home5                  auto   auto,defaults,noatime         0 0