0

Load is high.

This is my top

top - 04:08:52 up 15 days, 15:14,  1 user,  load average: 99.09, 107.09, 117.35
Tasks: 998 total, 125 running, 827 sleeping,   1 stopped,  45 zombie
Cpu(s): 13.4%us, 75.1%sy,  0.0%ni, 11.3%id,  0.0%wa,  0.0%hi,  0.2%si,  0.0%st
Mem:  65951648k total, 65390852k used,   560796k free,  5023756k buffers
Swap:  4194300k total,   262684k used,  3931616k free, 39846488k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
14904 nobody    20   0  151m  50m 2508 R 15.4  0.1  11:12.89 httpd
 1769 root      20   0     0    0    0 R 15.1  0.0 355:06.40 kondemand/22
 1778 root      20   0     0    0    0 S 14.8  0.0 345:52.47 kondemand/31
 1767 root      20   0     0    0    0 S 14.4  0.0 353:41.57 kondemand/20
 1766 root      20   0     0    0    0 S 13.8  0.0 374:52.13 kondemand/19
15188 newgames  20   0     0    0    0 Z 13.8  0.0   0:00.42 php <defunct>
 1772 root      20   0     0    0    0 R 13.4  0.0 337:16.18 kondemand/25
 1775 root      20   0     0    0    0 S 13.4  0.0 352:11.24 kondemand/28
11857 mailnull  20   0 71640 1408  752 S 12.8  0.0   0:39.47 exim
15180 hearsttr  20   0  179m  10m 6020 R 12.8  0.0   0:00.39 php
20980 nobody    20   0  151m  50m 2084 R 12.8  0.1  12:14.03 httpd
 1755 root      20   0     0    0    0 S 12.5  0.0 364:43.87 kondemand/8
 1764 root      20   0     0    0    0 R 12.5  0.0 342:25.04 kondemand/17
15161 hearsttr  20   0     0    0    0 Z 12.5  0.0   0:00.38 php <defunct>
 1756 root      20   0     0    0    0 R 12.1  0.0 352:29.80 kondemand/9
 1771 root      20   0     0    0    0 R 12.1  0.0 349:01.36 kondemand/24
15138 hearsttr  20   0     0    0    0 Z 12.1  0.0   0:00.37 php <defunct>
 1757 root      20   0     0    0    0 S 11.8  0.0 368:19.92 kondemand/10
 1773 root      20   0     0    0    0 S 11.8  0.0 355:11.04 kondemand/26
15174 newgames  20   0     0    0    0 Z 11.8  0.0   0:00.36 php <defunct>
15440 nobody    20   0  151m  50m 2080 R 11.8  0.1  10:13.40 httpd
 1760 root      20   0     0    0    0 R 11.5  0.0 368:08.78 kondemand/13
 1765 root      20   0     0    0    0 S 11.5  0.0 358:25.81 kondemand/18
15817 nobody    20   0  151m  50m 2112 R 11.5  0.1  12:15.94 httpd
15194 hearsttr  20   0  187m  12m 5820 S 11.1  0.0   0:00.34 php
15217 hearsttr  20   0     0    0    0 Z 10.8  0.0   0:00.33 php <defunct>
15226 newgames  20   0     0    0    0 Z 10.8  0.0   0:00.33 php <defunct>
 1219 nobody    20   0  151m  50m 2084 R 10.5  0.1   2:26.14 httpd
 1758 root      20   0     0    0    0 S 10.5  0.0 375:39.42 kondemand/11
 1759 root      20   0     0    0    0 R 10.5  0.0 364:42.21 kondemand/12
 1763 root      20   0     0    0    0 S 10.5  0.0 355:34.66 kondemand/16
 1770 root      20   0     0    0    0 S 10.5  0.0 350:08.17 kondemand/23
 6111 nobody    20   0  151m  50m 2508 R 10.5  0.1   9:23.07 httpd
14425 nobody    20   0  151m  50m 2208 S 10.5  0.1  11:37.53 httpd
15085 nobody    20   0  151m  50m 2508 R 10.5  0.1  11:11.37 httpd
15228 sexsmovi  20   0  188m  12m 5820 S 10.5  0.0   0:00.32 php
15378 nobody    20   0  151m  50m 2508 S 10.5  0.1  11:46.37 httpd
15522 nobody    20   0  151m  50m 2196 S 10.5  0.1  10:54.56 httpd
16446 nobody    20   0  151m  50m 2988 S 10.5  0.1  11:12.03 httpd
16962 nobody    20   0  151m  50m 2196 S 10.5  0.1  11:05.68 httpd
18266 nobody    20   0  151m  50m 2084 S 10.5  0.1  11:41.40 httpd
18334 nobody    20   0  151m  50m 2468 S 10.5  0.1  10:39.56 httpd
18774 nobody    20   0  151m  50m 2092 S 10.5  0.1  11:00.09 httpd
  817 nobody    20   0  151m  51m 2992 S 10.2  0.1  10:09.30 httpd
 1882 nobody    20   0  151m  50m 2844 S 10.2  0.1  11:40.74 httpd
14523 nobody    20   0  151m  50m 3000 S 10.2  0.1  11:30.28 httpd
14900 nobody    20   0  151m  50m 2080 S 10.2  0.1  10:38.68 httpd
14919 nobody    20   0  151m  50m 2540 S 10.2  0.1  11:33.35 httpd
15178 nobody    20   0  151m  50m 2508 S 10.2  0.1  11:35.48 httpd
15223 romanced  20   0     0    0    0 Z 10.2  0.0   0:00.31 php <defunct>
15251 nudeteen  20   0  184m 9.9m 5204 R 10.2  0.0   0:00.31 php
15259 investgr  20   0  184m 9884 5164 R 10.2  0.0   0:00.31 php
15460 nobody    20   0  151m  50m 2076 S 10.2  0.1  12:09.73 httpd
15944 nobody    20   0  151m  50m 2092 R 10.2  0.1  11:27.52 httpd
16168 nobody    20   0  151m  50m 2508 S 10.2  0.1  11:40.34 httpd
16736 nobody    20   0  151m  50m 2968 R 10.2  0.1  11:42.38 httpd
17367 nobody    20   0  151m  50m 2228 S 10.2  0.1  10:32.59 httpd
17438 nobody    20   0  151m  50m 2540 S 10.2  0.1  10:54.28 httpd
17882 nobody    20   0  151m  50m 2980 S 10.2  0.1  11:12.86 httpd

Notice that 75% of the CPU is used by system instead of users. I wonder what they do?

root@host [~]# iostat -xk 5
Linux 2.6.32-504.el6.x86_64 (host.buildingsuperteams.com)       01/14/2015      _x86_64_        (32 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           6.75    0.03   13.14    0.02    0.00   80.06

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.27   132.84    1.38   37.44    17.17  1371.08    71.52     0.42   10.77   0.69   2.67
sdb               0.05    38.16    0.51   29.32    12.26   273.27    19.14     0.19    6.32   0.32   0.94

iostat doesn't show any excessive usage of the ssd

running iftop doesn't seem show the net to be the bottle neck.

Something is running on system but what?

On my other server some suggest

Your server has a high load because of how you have Apache configured, as each new php access opens a new process.

I enable Apache keepalive for this vhost and it seems to be a lowering the load.

I wonder where I can learn more about it.

Here is some info on the 16 core cpu

root@host [~]# grep -E '^model name|^cpu MHz' /proc/cpuinfo
model name      : AMD Opteron(TM) Processor 6272
cpu MHz         : 2100.000
model name      : AMD Opteron(TM) Processor 6272
cpu MHz         : 2100.000
model name      : AMD Opteron(TM) Processor 6272
cpu MHz         : 2100.000
model name      : AMD Opteron(TM) Processor 6272
cpu MHz         : 2100.000
model name      : AMD Opteron(TM) Processor 6272
cpu MHz         : 2100.000
model name      : AMD Opteron(TM) Processor 6272
cpu MHz         : 2100.000
model name      : AMD Opteron(TM) Processor 6272
cpu MHz         : 2100.000
model name      : AMD Opteron(TM) Processor 6272
cpu MHz         : 2100.000
model name      : AMD Opteron(TM) Processor 6272
cpu MHz         : 2100.000
model name      : AMD Opteron(TM) Processor 6272
cpu MHz         : 2100.000
model name      : AMD Opteron(TM) Processor 6272
cpu MHz         : 2100.000
model name      : AMD Opteron(TM) Processor 6272
cpu MHz         : 2100.000
model name      : AMD Opteron(TM) Processor 6272
cpu MHz         : 2100.000
model name      : AMD Opteron(TM) Processor 6272
cpu MHz         : 2100.000
model name      : AMD Opteron(TM) Processor 6272
cpu MHz         : 2100.000
model name      : AMD Opteron(TM) Processor 6272
cpu MHz         : 1400.000
model name      : AMD Opteron(TM) Processor 6272
cpu MHz         : 2100.000
model name      : AMD Opteron(TM) Processor 6272
cpu MHz         : 2100.000
model name      : AMD Opteron(TM) Processor 6272
cpu MHz         : 2100.000
model name      : AMD Opteron(TM) Processor 6272
cpu MHz         : 2100.000
model name      : AMD Opteron(TM) Processor 6272
cpu MHz         : 2100.000
model name      : AMD Opteron(TM) Processor 6272
cpu MHz         : 2100.000
model name      : AMD Opteron(TM) Processor 6272
cpu MHz         : 2100.000
model name      : AMD Opteron(TM) Processor 6272
cpu MHz         : 2100.000
model name      : AMD Opteron(TM) Processor 6272
cpu MHz         : 2100.000
model name      : AMD Opteron(TM) Processor 6272
cpu MHz         : 2100.000
model name      : AMD Opteron(TM) Processor 6272
cpu MHz         : 2100.000
model name      : AMD Opteron(TM) Processor 6272
cpu MHz         : 2100.000
model name      : AMD Opteron(TM) Processor 6272
cpu MHz         : 2100.000
model name      : AMD Opteron(TM) Processor 6272
cpu MHz         : 2100.000
model name      : AMD Opteron(TM) Processor 6272
cpu MHz         : 2100.000
model name      : AMD Opteron(TM) Processor 6272
cpu MHz         : 2100.000

migrated from serverfault.com Jan 15 '15 at 4:36

This question came from our site for system and network administrators.

  • 1
    I would check if your CPU is being throttled (kondemand handles throttling and CPU on those seem pretty high). If you aren't sure how to get started on that, try this link. Using CPUfreq Governors might be helpful as well. – yoonix Jan 14 '15 at 3:49
  • cpu MHz : 1400.000 <-- Looks like it might be throttling. I'd disable it and check in your BIOS if you have a power management setting labelled something like 'Performance'. – yoonix Jan 14 '15 at 5:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.