How to determine why a machine is running slowly?

Question

I log into a linux server (installed as a virtual machine) using a graphical ssh client (securessh). This server runs a tomcat5.5 server where nexus is installed.

When I type commands or delete/copy small files (around 5-6 MB), the shell takes a long time to respond (from 10 seconds to almost a minute). I have tried to run top to see if any processes use a lot of memory/cpu time:

top - 13:34:41 up 86 days, 16:04,  1 user,  load average: 2.13, 0.99, 1.94
Tasks:  63 total,   1 running,  62 sleeping,   0 stopped,   0 zombie
Cpu(s):  2.0%us,  1.5%sy,  0.0%ni, 96.2%id,  0.2%wa,  0.0%hi,  0.1%si,  0.0%st
Mem:   3896416k total,  3097824k used,   798592k free,   167180k buffers
Swap:   915664k total,       84k used,   915580k free,  2409236k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                      
20436 tomcat55  20   0  359m 217m  13m S   18  5.7   2713:04 jsvc                                 

Only the tomcat55 user use a significant amount of resources. Based on the above output is seems that this user only spends 5.7% of the mem and only 5.7% of the cpu. Am I misreading top's output? Why is the machine performing so poorly if the CPU and memory are so underutilized?

---

Edit: I have now tried to run atop and get:

ATOP - repository         2011/09/20  16:08:48               10 seconds elapsed
PRC | sys   0.17s | user   0.03s | #proc     64 | #zombie    0 | #exit      4 |
CPU | sys      2% | user      1% | irq       0% | idle    198% | wait      0% |
cpu | sys      1% | user      1% | irq       0% | idle     98% | cpu001 w  0% |
cpu | sys      0% | user      0% | irq       0% | idle     99% | cpu000 w  0% |
CPL | avg1   0.05 | avg5    0.92 | avg15   1.29 | csw      976 | intr      61 |
MEM | tot    3.7G | free  656.7M | cache   2.4G | buff  170.9M | slab  241.3M |
SWP | tot  894.2M | free  894.1M |              | vmcom 781.9M | vmlim   2.7G |
DSK |         sda | busy      0% | read       0 | write      9 | avio    0 ms |
NET | transport   | tcpi      18 | tcpo      26 | udpi       0 | udpo       0 |
NET | network     | ipi       22 | ipo       26 | ipfrw      0 | deliv     22 |
NET | eth1     0% | pcki      34 | pcko      26 | si    2 Kbps | so   11 Kbps |

  PID  SYSCPU  USRCPU  VGROW  RGROW  RDDSK  WRDSK  ST EXC S  CPU CMD     1/1   
 4687   0.06s   0.02s     0K     0K      -      -  NE   0 E   1% <lsb_release>
 4689   0.04s   0.01s     0K     0K      -      -  NE   0 E   1% <apt-cache>   
 4684   0.04s   0.00s   132K   132K     0K     0K  --   - R   0% atop
 4673   0.02s   0.00s     0K     0K     0K     0K  --   - S   0% sshd         
 4152   0.01s   0.00s     0K     0K     0K     0K  --   - S   0% vmware-guestd
 2302   0.00s   0.00s     0K     0K     0K     4K  --   - S   0% kjournald
 4688   0.00s   0.00s     0K     0K      -      -  NE   0 E   0% <sh> 
 4686   0.00s   0.00s     0K     0K      -      -  NE   0 E   0% <sh>    

If I understand this correct there a no 'zombies' but still they take up most of the cpu time (it jumps from 199% to 200%). Is this expected behavior?

Answer 1

In addition to iostat you should also consider atop ( http://www.atoptool.nl/ ) to identify non-cpu bottlenecks.

Answer 2

The problem may be slow disk I/O. Try running:

$ iostat -d -x 5 3

There's an explanation of the output here

Answer 3

Filesystem commands such as cp shouldn't be taking much CPU time or memory at all, which is what top shows. Try the iotop program instead (you may have to install it).

Answer 4

Since you're logging into a virtual machine, the slowness could be from other VMs on the physical host. From inside your virtual machine you cannot, generally, tell how much CPU you really have, nor can you see stats about the actual physical IO rates. You need to run 'top' or 'atop' on the physical host.

Answer 5

I cannot find the source right now and for that I apologise, but I remember reading it can be related to where the virtual drive is stored. If it is stored and accessed over NFS then there is a huge amount of overhead and thrashing.

Virtualbox has options to tweak the IO cache which you may need to look at. This affects simple programs that copy data such as cp.