# How to interpret this top command

top - 04:36:16 up 32 days,  2:33,  1 user,  load average: 251.72, 250.54, 231.19
Tasks: 785 total, 249 running, 522 sleeping,   2 stopped,  12 zombie
Cpu(s):  8.9%us, 90.5%sy,  0.4%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.2%si,  0.0%st
Mem:  16313868k total,  7021336k used,  9292532k free,   432196k buffers
Swap:  4194296k total,   295012k used,  3899284k free,   514320k cached

PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
18873 nudenude  20   0 97.8m  26m 4944 R  4.0  0.2   0:09.35 php
18140 nudenude  20   0 93984  22m 4940 R  3.7  0.1   0:11.04 php
18178 nudenude  20   0 99.6m  26m 4912 R  3.7  0.2   0:11.11 php
18537 nudenude  20   0  100m  26m 4936 R  3.7  0.2   0:10.31 php
18726 nudenude  20   0 98.3m  25m 4868 R  3.7  0.2   0:09.82 php
18977 nudenude  20   0  101m  25m 4952 R  3.7  0.2   0:09.14 php
19049 nudenude  20   0 99.5m  28m 4920 R  3.7  0.2   0:08.87 php
19852 nudenude  20   0  100m  30m 4912 R  3.7  0.2   0:07.08 php
20504 nudenude  20   0 93024  18m 4880 R  3.7  0.1   0:04.86 php
20562 nudenude  20   0  100m  31m 4940 R  3.7  0.2   0:04.73 php
20681 nudenude  20   0 90912  14m 4940 R  3.7  0.1   0:04.40 php
20685 nudenude  20   0 90656  15m 4928 R  3.7  0.1   0:04.38 php


Some things are strange.

Why system utilize tons of CPU for %sy?

It seems to me the tasks use a lot of virtual memory even though there are still plenty of memory left. So I wonder why?

Also what does 15m, 14m means in res and VIRT. Nothing is shown in man top.

I checked http://unixhelp.ed.ac.uk/CGI/man-cgi?top and non of it is shown.

Note: the load used to be 50%. Moreover, column VIRT is usually 0. However, once in a while it goes to this stuck mode where 90% of CPU is used by kernel. Don't know what the hell is that kernel doing.

very little CPU is used by users. No wonder load is extremely high. But what? What is the kernel doing>

-
I'd be more worried about the load if I were you. Also, it's a common misconception that VM equals swap. –  schaiba Feb 16 '13 at 12:47
For virt and res, it is explained in that man page you link to. What are your scripts doing? (Interesting username.) –  Mat Feb 16 '13 at 13:34
You have 12 zombies and very high load which are things to worry about. –  Naai Sekar Feb 16 '13 at 14:17
A zombie is a process that has finished, whose parent hasn't, but the partent has not asked for the statistics of the dead child. Sloppy programming, a bad sign for sure (but nothing urgent to worry about). –  vonbrand Feb 16 '13 at 14:33
What does 92m, 31m, etc mean? Why so much cpu usage in kernel? –  Jim Thio Feb 16 '13 at 15:09

I'm copying this from a man page I wrote for plog, since I was trying to make it clear there:

It is important to understand the difference between virtual address space and physical memory in interpreting some of the above statistics. As the name implies, virtual address space is not real; it’s basically a map of all the memory currently allocated to a process. The limit on the size of this map is the same for each processes (generally, 2-4 GB), and it is not accumulated (ie, you may have dozens or hundreds of processes, each with its own 2-4 GB virtual address space, on a system that only actually has 512 MB of physical memory).

Data cannot actually be stored or retrieved from virtual address space; real data requires real, physical memory. It is the kernel’s job to manage one in relation to another. Virtual space stats (VirtualSz, Data+Stack, and Priv&Write) are useful for considering the structure of a process and the relationship to physical memory use, but with regard to amount of RAM actually used, the physical memory stats (ResidentSz, Share, and Proportion) are what counts.

Top doesn't quite have all those metrics, but the VIRT score is virtual address space, RES refers to physical memory as does SHR. If you are concerned about relative memory usage (ie, one process compared to another), the RES score is more relevant.

Certain parts of VIRT are relevant relative to other processes; visors such as openVZ limit containers based on the total amount of private writable address space, not RSS. Top doesn't report this, but pmap and plog do (see the plog manpage for "Priv&Write"; this was actually part of my motive when writing it).

-
Note that RES is the part in memory of the virtual address space, and lots of that will be shared among processes. –  vonbrand Feb 16 '13 at 14:30
@vonbrand It's not documented in man proc, but the (linux) kernel (as of 2.6.something?) reports a Pss figure in /proc/<pid>/smaps. This is like Rss (top's "RES") except it is just the unshared space plus the shared regions divided by the number of other processes sharing them, which is pretty handy. Unfortunately top doesn't report that (but plog does ;)). Simply subtracting SHR from RES isn't very accurate, because SHR is real. If there are only two processes accessing a library, that space is "shared", but not much. –  goldilocks Feb 16 '13 at 15:01
you know why %sy is so high>? –  Jim Thio Feb 16 '13 at 15:54
@Thio : no idea. The kernel is spinning its wheels over something. Try ps -A -o state,pid,comm | grep "^D". If you see anything, then you may have disk corruption, which will cause a system I/O busy loop (google "uninterruptible sleep"). That's all I can think of, not knowing anything else about the system itself. It's unlikely tho, as I think those D processes would be evident with high CPU usage in top. –  goldilocks Feb 16 '13 at 16:54

As per output of your top, it seems there is problem in your php scripts , you can check

• make sure php script is properly working or not
• check error logs of apache
• check cron entry some time many php script run through cron
• also try to find out reason for zombie processes
• tune apache and php

I am sure that high load because of php, you can simply kill php processes by executing

killall -9 php Or pkill -9 php then Observe in top.

-