Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have an Ubuntu (12.04) NFS server that has a high load (larger than 10) even when nothing is running.

In detail, the storage is provided by an iSCSI device, on which I have 5 logical volumes (LVM) and some ext4 partitions. Even with all services stopped, and no exports (that is no client traffic), the load is at 10. Running iostat shows that one particular mapped device (/dev/dm-1) is always being written to (if I interpret the output correctly):

Device:            tps    MB_read/s    MB_wrtn/s    MB_read    MB_wrtn
sda               0.00         0.00         0.00          0          0
sdb             342.00         0.00         9.21          0          9
dm-0              0.00         0.00         0.00          0          0
dm-1            615.00         0.00        11.71          0         11

(sdb is where the iSCSI device appears, dm-n the various logical volumes). I have literally stopped (almost) all other services running, and I can with good confidence say that the moment I start the NFS server the load goes up, and when I stop it the load goes down. What is going on? How can I see what is writing to the disk? (I tried lsof but that shows no process).

Additions Adding information as asked.

uptime says:

18:27:15 up 1 day,  9:59,  2 users,  load average: 14.22, 12.42, 11.55

vmstat says:

procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 1  1    960 865924  51604 23204424    0    0    43     4    7   20  0  2 86 12

mpstat says:

Linux 3.2.0-26-generic-pae (leitrim)    12/25/2012  _i686_  (8 CPU)

06:33:53 PM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
06:33:53 PM  all    0.08    0.04    0.96   11.62    0.00    0.88    0.00    0.00   86.43

dstat -cdD sdb -ng 60 3

----total-cpu-usage---- --dsk/sdb-- -net/total- ---paging--
usr sys idl wai hiq siq| read  writ| recv  send|  in   out 
  0   1  86  12   0   1| 324k 8729k|   0     0 |   1B    8B
  0   1  87  12   0   1|  17k 8953k|8064k 9652k|   0     0 
  0   1  86  12   0   1|1229B 9081k|8010k 9796k|   0    68B
  0   1  89  10   0   1|3209B 8364k|7703k 9014k|   0     0 

rpcinfo

  program version netid     address                service    owner
    100000    4    tcp6      ::.0.111               portmapper superuser
    100000    3    tcp6      ::.0.111               portmapper superuser
    100000    4    udp6      ::.0.111               portmapper superuser
    100000    3    udp6      ::.0.111               portmapper superuser
    100000    4    tcp       0.0.0.0.0.111          portmapper superuser
    100000    3    tcp       0.0.0.0.0.111          portmapper superuser
    100000    2    tcp       0.0.0.0.0.111          portmapper superuser
    100000    4    udp       0.0.0.0.0.111          portmapper superuser
    100000    3    udp       0.0.0.0.0.111          portmapper superuser
    100000    2    udp       0.0.0.0.0.111          portmapper superuser
    100000    4    local     /run/rpcbind.sock      portmapper superuser
    100000    3    local     /run/rpcbind.sock      portmapper superuser
    100024    1    udp       0.0.0.0.172.13         status     116
    100024    1    tcp       0.0.0.0.229.210        status     116
    100024    1    udp6      ::.137.98              status     116
    100024    1    tcp6      ::.175.197             status     116
    100021    1    udp       0.0.0.0.171.56         nlockmgr   superuser
    100021    3    udp       0.0.0.0.171.56         nlockmgr   superuser
    100021    4    udp       0.0.0.0.171.56         nlockmgr   superuser
    100021    1    tcp       0.0.0.0.153.54         nlockmgr   superuser
    100021    3    tcp       0.0.0.0.153.54         nlockmgr   superuser
    100021    4    tcp       0.0.0.0.153.54         nlockmgr   superuser
    100021    1    udp6      ::.206.206             nlockmgr   superuser
    100021    3    udp6      ::.206.206             nlockmgr   superuser
    100021    4    udp6      ::.206.206             nlockmgr   superuser
    100021    1    tcp6      ::.132.23              nlockmgr   superuser
    100021    3    tcp6      ::.132.23              nlockmgr   superuser
    100021    4    tcp6      ::.132.23              nlockmgr   superuser
    100003    2    tcp       0.0.0.0.8.1            nfs        superuser
    100003    3    tcp       0.0.0.0.8.1            nfs        superuser
    100227    2    tcp       0.0.0.0.8.1            -          superuser
    100227    3    tcp       0.0.0.0.8.1            -          superuser
    100003    2    udp       0.0.0.0.8.1            nfs        superuser
    100003    3    udp       0.0.0.0.8.1            nfs        superuser
    100227    2    udp       0.0.0.0.8.1            -          superuser
    100227    3    udp       0.0.0.0.8.1            -          superuser
    100003    2    tcp6      ::.8.1                 nfs        superuser
    100003    3    tcp6      ::.8.1                 nfs        superuser
    100227    2    tcp6      ::.8.1                 -          superuser
    100227    3    tcp6      ::.8.1                 -          superuser
    100003    2    udp6      ::.8.1                 nfs        superuser
    100003    3    udp6      ::.8.1                 nfs        superuser
    100227    2    udp6      ::.8.1                 -          superuser
    100227    3    udp6      ::.8.1                 -          superuser
    100005    1    udp       0.0.0.0.154.132        mountd     superuser
    100005    1    tcp       0.0.0.0.217.173        mountd     superuser
    100005    1    udp6      ::.165.76              mountd     superuser
    100005    1    tcp6      ::.141.19              mountd     superuser
    100005    2    udp       0.0.0.0.183.55         mountd     superuser
    100005    2    tcp       0.0.0.0.214.84         mountd     superuser
    100005    2    udp6      ::.233.222             mountd     superuser
    100005    2    tcp6      ::.211.16              mountd     superuser
    100005    3    udp       0.0.0.0.188.56         mountd     superuser
    100005    3    tcp       0.0.0.0.158.62         mountd     superuser
    100005    3    udp6      ::.152.158             mountd     superuser
    100005    3    tcp6      ::.201.200             mountd     superuser
share|improve this question
    
Please add output of uptime, vmstat, mpstat. –  John Siu Dec 25 '12 at 6:16
    
What is the output for dstat -cdD sdb -ng 60 3? Wait three minutes for the command to complete. –  jippie Dec 25 '12 at 11:36
    
@JohnSiu I edited my question to add the info you requested. Thanks –  Giuseppe Vacanti Dec 25 '12 at 17:42
    
@jippie added the info you wanted. Thanks. –  Giuseppe Vacanti Dec 25 '12 at 17:43
    
@GiuseppeVacanti add cat /proc/mdstate. I wonder if it is doing disk sync. Did you change disk or have a system crash recently? –  John Siu Dec 25 '12 at 19:03
show 3 more comments

1 Answer

After some more investigation I can answer my own questions, at least partially:

  1. Writing to one of the devices was being caused by a process running wild on one of the NFS clients. I got there by shutting down one client at a time. It would be nice to have a command to see which client is writing to the NFS server, but if there's one I could not find it.
share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.