1
  • Debian 8
  • 1 minute average server load: 39

In that very moment, I ran several commands and save the output to files (which I present here).

What's wrong? This very high load occurs randomly several times a week:

top -b -n 1 | head -n 20

top - 11:07:25 up 25 days,  1:57,  1 user,  load average: 39,14, 11,17, 4,16
Tasks: 148 total,  43 running, 104 sleeping,   0 stopped,   1 zombie
%Cpu(s): 11,0 us,  2,3 sy,  0,0 ni, 86,2 id,  0,1 wa,  0,0 hi,  0,3 si,  0,0 st
KiB Mem:   3853724 total,  2984252 used,   869472 free,    33592 buffers
KiB Swap:        0 total,        0 used,        0 free,   225136 cached

  PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND
23748 www-data  20   0  811m  73m  33m R   6,2  1,9   0:02.32 apache2
23819 www-data  20   0  593m  70m  32m R   6,2  1,9   0:01.41 apache2
23820 www-data  20   0  810m 112m  33m S   6,2  3,0   0:01.94 apache2
23891 www-data  20   0  459m  67m  29m R   6,2  1,8   0:01.12 apache2
 1201 root      20   0     0    0    0 S   3,1  0,0   5:57.44 jbd2/xvdb1-8
23708 www-data  20   0  670m  76m  35m R   3,1  2,0   0:03.63 apache2
23738 www-data  20   0  740m  79m  37m R   3,1  2,1   0:04.85 apache2
23783 www-data  20   0  740m  83m  34m R   3,1  2,2   0:02.54 apache2
23796 www-data  20   0  596m  90m  33m R   3,1  2,4   0:02.10 apache2
23797 www-data  20   0  670m  73m  33m R   3,1  2,0   0:02.13 apache2
23805 www-data  20   0  595m  91m  34m R   3,1  2,4   0:01.70 apache2
23806 www-data  20   0  816m 104m  32m S   3,1  2,8   0:01.87 apache2
23807 www-data  20   0  812m  74m  35m R   3,1  2,0   0:02.03 apache2    

free -m

             total       used       free     shared    buffers     cached
Mem:          3763       2946        817          0         33        221
-/+ buffers/cache:       2691       1071
Swap:            0          0          0

iostat

Linux 3.2.0-4-amd64 (sntc)  29/10/17    _x86_64_    (2 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          10,99    0,00    2,66    0,13    0,01   86,21

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
xvdap1           12,40        17,30       198,41   37492508  429964572
xvdb             12,33        22,50       459,16   48759538  995030028
xvdc              0,60        12,55        13,34   27186977   28917504
xvdf              5,52        36,06        56,61   78147255  122670146

ps -e v

  PID TTY      STAT   TIME  MAJFL   TRS   DRS   RSS %MEM COMMAND
    1 ?        Ss     0:16     52    32 15447   724  0.0 init [2]         
    2 ?        S      0:00      0     0     0     0  0.0 [kthreadd]
    3 ?        S      0:53      0     0     0     0  0.0 [ksoftirqd/0]
    5 ?        S      0:00      0     0     0     0  0.0 [kworker/u:0]
    6 ?        S      0:02      0     0     0     0  0.0 [migration/0]
    7 ?        S      0:06      0     0     0     0  0.0 [watchdog/0]
    8 ?        S      0:02      0     0     0     0  0.0 [migration/1]
    9 ?        S      0:00      0     0     0     0  0.0 [kworker/1:0]
   10 ?        S      0:47      0     0     0     0  0.0 [ksoftirqd/1]
   11 ?        S      0:05      0     0     0     0  0.0 [watchdog/1]
   12 ?        S<     0:00      0     0     0     0  0.0 [cpuset]
   13 ?        S<     0:00      0     0     0     0  0.0 [khelper]
   14 ?        S      0:00      0     0     0     0  0.0 [kdevtmpfs]
   15 ?        S<     0:00      0     0     0     0  0.0 [netns]
   16 ?        S      0:00      0     0     0     0  0.0 [xenwatch]
   17 ?        S      0:00      0     0     0     0  0.0 [xenbus]
   18 ?        S      0:04      0     0     0     0  0.0 [sync_supers]
   19 ?        S      0:00      0     0     0     0  0.0 [bdi-default]
   20 ?        S<     0:00      0     0     0     0  0.0 [kintegrityd]
   21 ?        S<     0:00      0     0     0     0  0.0 [kblockd]
   22 ?        S      2:11      0     0     0     0  0.0 [kworker/1:1]
   23 ?        S      0:00      0     0     0     0  0.0 [khungtaskd]
   24 ?        S      1:03      0     0     0     0  0.0 [kswapd0]
   25 ?        SN     0:00      0     0     0     0  0.0 [ksmd]
   26 ?        S      0:00      0     0     0     0  0.0 [fsnotify_mark]
   27 ?        S<     0:00      0     0     0     0  0.0 [crypto]
   30 ?        S      0:00      0     0     0     0  0.0 [khvcd]
  115 ?        S      1:47      0     0     0     0  0.0 [jbd2/xvda1-8]
  116 ?        S<     0:00      0     0     0     0  0.0 [ext4-dio-unwrit]
  256 ?        Ss     0:00      2   143 23728   376  0.0 udevd --daemon
  338 ?        S      0:00      0     0     0     0  0.0 [kworker/u:1]
 1201 ?        S      5:57      0     0     0     0  0.0 [jbd2/xvdb1-8]
 1202 ?        S<     0:00      0     0     0     0  0.0 [ext4-dio-unwrit]
 1203 ?        S      0:02      0     0     0     0  0.0 [jbd2/xvdc1-8]
 1204 ?        S<     0:00      0     0     0     0  0.0 [ext4-dio-unwrit]
 1205 ?        S      1:29      0     0     0     0  0.0 [jbd2/xvdf5-8]
 1206 ?        S<     0:00      0     0     0     0  0.0 [ext4-dio-unwrit]
 1207 ?        S      0:37      0     0     0     0  0.0 [jbd2/xvdf6-8]
 1208 ?        S<     0:00      0     0     0     0  0.0 [ext4-dio-unwrit]
 1209 ?        S      0:00      0     0     0     0  0.0 [jbd2/xvdf7-8]
 1210 ?        S<     0:00      0     0     0     0  0.0 [ext4-dio-unwrit]
 1211 ?        S      0:00      0     0     0     0  0.0 [jbd2/xvdf8-8]
 1212 ?        S<     0:00      0     0     0     0  0.0 [ext4-dio-unwrit]
 1213 ?        S      0:00      0     0     0     0  0.0 [jbd2/xvdf9-8]
 1214 ?        S<     0:00      0     0     0     0  0.0 [ext4-dio-unwrit]
 1447 ?        Ss     0:00     26    64  8447   152  0.0 /sbin/dhcpcd-bin -h sntc -Y -N eth0
 1585 ?        S      0:03      0     0     0     0  0.0 [kworker/0:1]
 1594 ?        Sl     2:13    336   346 249749 5212  0.1 /usr/sbin/rsyslogd -c5
 1666 ?        S      2:07      0     0     0     0  0.0 [flush-202:1]
 1667 ?        S     13:58      0     0     0     0  0.0 [flush-202:16]
 1669 ?        S      0:19      0     0     0     0  0.0 [flush-202:80]
 1733 ?        Ss     0:00     13    18 16781   164  0.0 /usr/sbin/atd
 1852 ?        Ss     1:16    243   494 32877   624  0.0 /usr/sbin/ntpd -p /var/run/ntpd.pid -g -u 102:104
 1950 ?        Sl     4:45   1165  2392 195875 6380  0.1 /usr/bin/python /usr/bin/fail2ban-server -b -s /var/run/fail2ban/fail2ban.sock
 1984 ?        S      0:42    141    84 22543   572  0.0 /usr/lib/gamin/gam_server
 4507 ?        S      0:00      0   109  4218   652  0.0 /bin/sh /usr/bin/mysqld_safe
 4599 ?        Ss     0:26     14    38 25245   416  0.0 /usr/sbin/cron
 4854 ?        Sl    17:48    305 10472 1337691 205796  5.3 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --user=mysql --log-error=/var/log/mysql/error.log --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sock --port=3306
 6594 ?        Sl     0:46    131    93 329398  500  0.0 /usr/bin/memcached -m 512 -p 11211 -u nobody -l 127.0.0.1
 9452 ?        S<L    0:01      1   149 17622  5500  0.1 /usr/bin/atop -a -w /var/log/atop/atop_20171029 600
15494 ?        Ss     2:16     76   618 441641 51568  1.3 /usr/sbin/apache2 -k start
15803 ?        Ss     0:05      3    33 36122  1412  0.0 /usr/lib/postfix/master
15807 ?        S      0:01      6    57 38322  1688  0.0 qmgr -l -t fifo -u
17826 ?        Ss     0:00      3   492 52211  1188  0.0 /usr/sbin/sshd
19834 ?        S      0:00      0     0     0     0  0.0 [kworker/0:0]
20494 ?        S      0:00     16    15 42432  1160  0.0 tlsmgr -l -t unix -u -c
23617 ?        Sl     0:04     20   618 740613 60132  1.5 /usr/sbin/apache2 -k start
23635 ?        S      0:00      0    10 38209  1132  0.0 anvil -l -t unix -u -c
23699 ?        Sl     0:04      8   618 820581 88384  2.2 /usr/sbin/apache2 -k start
23708 ?        Rl     0:03     10   618 686101 78032  2.0 /usr/sbin/apache2 -k start
23738 ?        Rl     0:04     10   618 758125 80936  2.1 /usr/sbin/apache2 -k start
23740 ?        Rl     0:03      5   618 610873 77436  2.0 /usr/sbin/apache2 -k start
23748 ?        Rl     0:02      7   618 830121 74928  1.9 /usr/sbin/apache2 -k start
23777 ?        S      0:00      2     9 38210  1148  0.0 pickup -l -t fifo -u -c
23783 ?        Rl     0:02     18   618 757709 85956  2.2 /usr/sbin/apache2 -k start
23787 ?        Sl     0:02      6   618 683949 76724  1.9 /usr/sbin/apache2 -k start
23788 ?        Rl     0:02     11   618 833877 98932  2.5 /usr/sbin/apache2 -k start
23793 ?        Rl     0:02     13   618 681797 72728  1.8 /usr/sbin/apache2 -k start
23795 ?        Sl     0:02     15   618 760273 78120  2.0 /usr/sbin/apache2 -k start
23796 ?        Sl     0:02      8   618 610017 93136  2.4 /usr/sbin/apache2 -k start
23797 ?        Rl     0:02      6   618 686101 75704  1.9 /usr/sbin/apache2 -k start
23802 ?        Sl     0:02     43   618 682725 124560  3.2 /usr/sbin/apache2 -k start
23805 ?        Rl     0:01      3   618 608885 93768  2.4 /usr/sbin/apache2 -k start
23806 ?        Sl     0:01     14   618 835453 107016  2.7 /usr/sbin/apache2 -k start
23807 ?        Sl     0:02     41   618 831877 76600  1.9 /usr/sbin/apache2 -k start
23808 ?        Rl     0:01     13   618 756437 92128  2.3 /usr/sbin/apache2 -k start
23810 ?        Sl     0:01      9   618 609357 74208  1.9 /usr/sbin/apache2 -k start
23817 ?        Rl     0:02      3   618 825697 81764  2.1 /usr/sbin/apache2 -k start
23819 ?        Rl     0:01      6   618 607533 72188  1.8 /usr/sbin/apache2 -k start
23820 ?        Sl     0:01     10   618 829533 115084  2.9 /usr/sbin/apache2 -k start
23821 ?        Sl     0:01      7   618 591217 52444  1.3 /usr/sbin/apache2 -k start
23823 ?        Rl     0:01     10   618 608889 92152  2.3 /usr/sbin/apache2 -k start
23824 ?        Z      0:01      6     0     0     0  0.0 [apache2] <defunct>
23827 ?        R      0:01      8   618 466189 68584  1.7 /usr/sbin/apache2 -k start
23828 ?        Rl     0:01     19   618 757573 93540  2.4 /usr/sbin/apache2 -k start
23829 ?        Sl     0:01     30   618 684861 78796  2.0 /usr/sbin/apache2 -k start
23830 ?        Rl     0:01     10   618 608889 91908  2.3 /usr/sbin/apache2 -k start
23832 ?        Sl     0:01     10   618 683041 64844  1.6 /usr/sbin/apache2 -k start
23833 ?        Rl     0:01      4   618 756401 94052  2.4 /usr/sbin/apache2 -k start
23835 ?        Rl     0:01      7   618 744637 64920  1.6 /usr/sbin/apache2 -k start
23843 ?        Rl     0:01      9   618 607557 90052  2.3 /usr/sbin/apache2 -k start
23844 ?        Sl     0:01      0   618 682501 74080  1.9 /usr/sbin/apache2 -k start
23847 ?        Sl     0:01      9   618 830045 79984  2.0 /usr/sbin/apache2 -k start
23848 ?        Sl     0:01      7   618 608637 91384  2.3 /usr/sbin/apache2 -k start
23849 ?        R      0:01      4   618 467569 71308  1.8 /usr/sbin/apache2 -k start
23850 ?        Rl     0:01      5   618 683589 72724  1.8 /usr/sbin/apache2 -k start
23854 ?        R      0:01      6   618 469693 68732  1.7 /usr/sbin/apache2 -k start
23855 ?        Rl     0:01      5   618 608837 74208  1.9 /usr/sbin/apache2 -k start
23856 ?        Rl     0:01      3   618 607517 71364  1.8 /usr/sbin/apache2 -k start
23857 ?        R      0:01      2   618 469869 70996  1.8 /usr/sbin/apache2 -k start
23858 ?        Sl     0:01      4   618 611705 76488  1.9 /usr/sbin/apache2 -k start
23860 ?        Sl     0:01      3   618 681249 74212  1.9 /usr/sbin/apache2 -k start
23862 ?        Rl     0:01      2   618 755493 83120  2.1 /usr/sbin/apache2 -k start
23864 ?        Sl     0:00      7   618 607529 71348  1.8 /usr/sbin/apache2 -k start
23870 ?        R      0:01     15   618 469161 69868  1.8 /usr/sbin/apache2 -k start
23872 ?        Sl     0:01     18   618 537669 81988  2.1 /usr/sbin/apache2 -k start
23873 ?        R      0:01      4   618 471633 71352  1.8 /usr/sbin/apache2 -k start
23874 ?        Sl     0:01      5   618 609809 85048  2.2 /usr/sbin/apache2 -k start
23875 ?        Rl     0:01     29   618 680925 69496  1.8 /usr/sbin/apache2 -k start
23876 ?        Rl     0:01      2   618 833901 77692  2.0 /usr/sbin/apache2 -k start
23882 ?        R      0:00     14   618 462697 61192  1.5 /usr/sbin/apache2 -k start
23885 ?        R      0:01      3   618 471633 71408  1.8 /usr/sbin/apache2 -k start
23889 ?        Rl     0:01      2   618 607797 71892  1.8 /usr/sbin/apache2 -k start
23891 ?        R      0:01      8   618 469409 69152  1.7 /usr/sbin/apache2 -k start
23894 ?        Sl     0:01      6   618 831525 78024  2.0 /usr/sbin/apache2 -k start
23895 ?        R      0:00      0   618 467553 68940  1.7 /usr/sbin/apache2 -k start
23896 ?        R      0:01     28   618 469789 70852  1.8 /usr/sbin/apache2 -k start
23897 ?        Sl     0:00     25   618 664033 50848  1.3 /usr/sbin/apache2 -k start
23908 ?        Sl     0:00      8   618 608837 74644  1.9 /usr/sbin/apache2 -k start
23911 ?        R      0:00      5   618 461933 62404  1.6 /usr/sbin/apache2 -k start
23912 ?        Sl     0:00      0   618 536837 87128  2.2 /usr/sbin/apache2 -k start
23918 ?        R      0:00      4   618 450233 40728  1.0 /usr/sbin/apache2 -k start
23920 ?        R      0:00      2   618 469321 72120  1.8 /usr/sbin/apache2 -k start
23922 ?        Rl     0:00      7   618 606693 69520  1.8 /usr/sbin/apache2 -k start
23923 ?        Rl     0:00      8   618 600745 68700  1.7 /usr/sbin/apache2 -k start
23957 ?        S      0:00     18    38 37909  1220  0.0 /USR/SBIN/CRON
23958 ?        Ss     0:00      3   109  4218   644  0.0 /bin/sh -c    /var/scripts/arregla_permisos_www.sh >/dev/null 2>&1
23963 ?        S      0:00     18   966 13469  1228  0.0 /bin/bash /var/scripts/arregla_permisos_www.sh
23968 ?        R      0:00      1    50  7049   816  0.0 chmod -R 770 /var/www/
23980 ?        Ss     0:00     10   492 71463  3684  0.0 sshd: root@pts/0 
23982 ?        S      0:00      2    21 38210  2344  0.0 trivial-rewrite -n rewrite -t unix -u -c
23984 ?        S      0:00      4   104 42443  2672  0.0 smtp -t unix -u -c
23985 ?        S      0:00      0   104 42443  2672  0.0 smtp -t unix -u -c
23988 ?        S      0:00      1    40 38215  2356  0.0 bounce -z -n defer -t unix -u -c
23990 pts/0    Ss     0:00      2   966 23365  3456  0.0 -bash
24085 pts/0    S+     0:00      0   966 13477  1452  0.0 /bin/bash /var/scripts/diagnosis_instantanea.sh
24092 pts/0    R+     0:00      1    84  9763   868  0.0 ps -e v

ps auxwwwf

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         2  0.0  0.0      0     0 ?        S    oct04   0:00 [kthreadd]
root         3  0.0  0.0      0     0 ?        S    oct04   0:53  \_ [ksoftirqd/0]
root         5  0.0  0.0      0     0 ?        S    oct04   0:00  \_ [kworker/u:0]
root         6  0.0  0.0      0     0 ?        S    oct04   0:02  \_ [migration/0]
root         7  0.0  0.0      0     0 ?        S    oct04   0:06  \_ [watchdog/0]
root         8  0.0  0.0      0     0 ?        S    oct04   0:02  \_ [migration/1]
root         9  0.0  0.0      0     0 ?        S    oct04   0:00  \_ [kworker/1:0]
root        10  0.0  0.0      0     0 ?        S    oct04   0:47  \_ [ksoftirqd/1]
root        11  0.0  0.0      0     0 ?        S    oct04   0:05  \_ [watchdog/1]
root        12  0.0  0.0      0     0 ?        S<   oct04   0:00  \_ [cpuset]
root        13  0.0  0.0      0     0 ?        S<   oct04   0:00  \_ [khelper]
root        14  0.0  0.0      0     0 ?        S    oct04   0:00  \_ [kdevtmpfs]
root        15  0.0  0.0      0     0 ?        S<   oct04   0:00  \_ [netns]
root        16  0.0  0.0      0     0 ?        S    oct04   0:00  \_ [xenwatch]
root        17  0.0  0.0      0     0 ?        S    oct04   0:00  \_ [xenbus]
root        18  0.0  0.0      0     0 ?        S    oct04   0:04  \_ [sync_supers]
root        19  0.0  0.0      0     0 ?        S    oct04   0:00  \_ [bdi-default]
root        20  0.0  0.0      0     0 ?        S<   oct04   0:00  \_ [kintegrityd]
root        21  0.0  0.0      0     0 ?        S<   oct04   0:00  \_ [kblockd]
root        22  0.0  0.0      0     0 ?        S    oct04   2:11  \_ [kworker/1:1]
root        23  0.0  0.0      0     0 ?        S    oct04   0:00  \_ [khungtaskd]
root        24  0.0  0.0      0     0 ?        S    oct04   1:03  \_ [kswapd0]
root        25  0.0  0.0      0     0 ?        SN   oct04   0:00  \_ [ksmd]
root        26  0.0  0.0      0     0 ?        S    oct04   0:00  \_ [fsnotify_mark]
root        27  0.0  0.0      0     0 ?        S<   oct04   0:00  \_ [crypto]
root        30  0.0  0.0      0     0 ?        S    oct04   0:00  \_ [khvcd]
root       115  0.0  0.0      0     0 ?        S    oct04   1:47  \_ [jbd2/xvda1-8]
root       116  0.0  0.0      0     0 ?        S<   oct04   0:00  \_ [ext4-dio-unwrit]
root       338  0.0  0.0      0     0 ?        S    oct04   0:00  \_ [kworker/u:1]
root      1201  0.0  0.0      0     0 ?        S    oct04   5:57  \_ [jbd2/xvdb1-8]
root      1202  0.0  0.0      0     0 ?        S<   oct04   0:00  \_ [ext4-dio-unwrit]
root      1203  0.0  0.0      0     0 ?        S    oct04   0:02  \_ [jbd2/xvdc1-8]
root      1204  0.0  0.0      0     0 ?        S<   oct04   0:00  \_ [ext4-dio-unwrit]
root      1205  0.0  0.0      0     0 ?        S    oct04   1:29  \_ [jbd2/xvdf5-8]
root      1206  0.0  0.0      0     0 ?        S<   oct04   0:00  \_ [ext4-dio-unwrit]
root      1207  0.0  0.0      0     0 ?        S    oct04   0:37  \_ [jbd2/xvdf6-8]
root      1208  0.0  0.0      0     0 ?        S<   oct04   0:00  \_ [ext4-dio-unwrit]
root      1209  0.0  0.0      0     0 ?        S    oct04   0:00  \_ [jbd2/xvdf7-8]
root      1210  0.0  0.0      0     0 ?        S<   oct04   0:00  \_ [ext4-dio-unwrit]
root      1211  0.0  0.0      0     0 ?        S    oct04   0:00  \_ [jbd2/xvdf8-8]
root      1212  0.0  0.0      0     0 ?        S<   oct04   0:00  \_ [ext4-dio-unwrit]
root      1213  0.0  0.0      0     0 ?        S    oct04   0:00  \_ [jbd2/xvdf9-8]
root      1214  0.0  0.0      0     0 ?        S<   oct04   0:00  \_ [ext4-dio-unwrit]
root      1666  0.0  0.0      0     0 ?        S    oct04   2:07  \_ [flush-202:1]
root      1667  0.0  0.0      0     0 ?        S    oct04  13:58  \_ [flush-202:16]
root      1669  0.0  0.0      0     0 ?        S    oct04   0:19  \_ [flush-202:80]
root      1585  0.0  0.0      0     0 ?        S    oct28   0:03  \_ [kworker/0:1]
root     19834  0.0  0.0      0     0 ?        S    09:35   0:00  \_ [kworker/0:0]
root         1  0.0  0.0  15480   724 ?        Ss   oct04   0:16 init [2]         
root       256  0.0  0.0  23872   376 ?        Ss   oct04   0:00 udevd --daemon
root      1447  0.0  0.0   8512   152 ?        Ss   oct04   0:00 /sbin/dhcpcd-bin -h sntc -Y -N eth0
root      1594  0.0  0.1 250096  5212 ?        Sl   oct04   2:13 /usr/sbin/rsyslogd -c5
daemon    1733  0.0  0.0  16800   164 ?        Ss   oct04   0:00 /usr/sbin/atd
ntp       1852  0.0  0.0  33372   624 ?        Ss   oct04   1:16 /usr/sbin/ntpd -p /var/run/ntpd.pid -g -u 102:104
root      1950  0.0  0.1 198268  6380 ?        Sl   oct04   4:45 /usr/bin/python /usr/bin/fail2ban-server -b -s /var/run/fail2ban/fail2ban.sock
root      1984  0.0  0.0  22628   572 ?        S    oct04   0:42 /usr/lib/gamin/gam_server
root      4599  0.0  0.0  25284   416 ?        Ss   oct04   0:26 /usr/sbin/cron
root     23957  0.0  0.0  37948  1220 ?        S    11:07   0:00  \_ /USR/SBIN/CRON
root     23958  0.0  0.0   4328   644 ?        Ss   11:07   0:00      \_ /bin/sh -c    /var/scripts/arregla_permisos_www.sh >/dev/null 2>&1
root     23963  0.0  0.0  14436  1228 ?        S    11:07   0:00          \_ /bin/bash /var/scripts/arregla_permisos_www.sh
root     23968  2.9  0.0   7100   816 ?        R    11:07   0:00              \_ chmod -R 770 /var/www/
nobody    6594  0.0  0.0 329492   500 ?        Sl   oct10   0:46 /usr/bin/memcached -m 512 -p 11211 -u nobody -l 127.0.0.1
root      4507  0.0  0.0   4328   652 ?        S    oct27   0:00 /bin/sh /usr/bin/mysqld_safe
mysql     4854  0.6  5.3 1348164 205860 ?      Sl   oct27  17:48  \_ /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --user=mysql --log-error=/var/log/mysql/error.log --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sock --port=3306
root     15494  0.0  1.3 442260 51568 ?        Ss   oct27   2:16 /usr/sbin/apache2 -k start
www-data 23617  2.5  1.5 741232 60132 ?        Sl   11:04   0:04  \_ /usr/sbin/apache2 -k start
www-data 23699  3.9  2.2 821200 88384 ?        Sl   11:05   0:04  \_ /usr/sbin/apache2 -k start
www-data 23708  2.9  1.7 674688 66688 ?        Rl   11:05   0:03  \_ /usr/sbin/apache2 -k start
www-data 23738  4.9  2.1 758744 80936 ?        Rl   11:05   0:04  \_ /usr/sbin/apache2 -k start
www-data 23740  3.3  2.0 611488 77436 ?        Rl   11:05   0:03  \_ /usr/sbin/apache2 -k start
www-data 23748  2.4  1.9 830740 74928 ?        Rl   11:05   0:02  \_ /usr/sbin/apache2 -k start
www-data 23783  4.2  2.2 758328 85956 ?        Rl   11:06   0:02  \_ /usr/sbin/apache2 -k start
www-data 23787  3.6  1.9 684568 76724 ?        Sl   11:06   0:02  \_ /usr/sbin/apache2 -k start
www-data 23788  4.1  2.5 834496 98932 ?        Rl   11:06   0:02  \_ /usr/sbin/apache2 -k start
www-data 23793  3.5  1.8 682416 72728 ?        Rl   11:06   0:02  \_ /usr/sbin/apache2 -k start
www-data 23795  3.5  2.0 760892 78120 ?        Sl   11:06   0:02  \_ /usr/sbin/apache2 -k start
www-data 23796  3.6  2.4 610636 93136 ?        Sl   11:06   0:02  \_ /usr/sbin/apache2 -k start
www-data 23797  3.6  1.9 686720 75704 ?        Rl   11:06   0:02  \_ /usr/sbin/apache2 -k start
www-data 23802  3.9  3.2 683344 124560 ?       Sl   11:06   0:02  \_ /usr/sbin/apache2 -k start
www-data 23805  2.9  2.4 609504 93768 ?        Rl   11:06   0:01  \_ /usr/sbin/apache2 -k start
www-data 23806  3.3  2.0 622616 79256 ?        Sl   11:06   0:01  \_ /usr/sbin/apache2 -k start
www-data 23807  3.5  1.9 832496 76600 ?        Sl   11:06   0:02  \_ /usr/sbin/apache2 -k start
www-data 23808  2.9  2.3 757056 92128 ?        Rl   11:06   0:01  \_ /usr/sbin/apache2 -k start
www-data 23810  2.1  1.9 609976 74208 ?        Sl   11:06   0:01  \_ /usr/sbin/apache2 -k start
www-data 23817  3.5  2.1 826828 82556 ?        Rl   11:06   0:02  \_ /usr/sbin/apache2 -k start
www-data 23819  2.5  1.8 608152 72188 ?        Rl   11:06   0:01  \_ /usr/sbin/apache2 -k start
www-data 23820  3.4  2.4 830152 96148 ?        Sl   11:06   0:01  \_ /usr/sbin/apache2 -k start
www-data 23821  3.0  1.3 591836 52444 ?        Sl   11:06   0:01  \_ /usr/sbin/apache2 -k start
www-data 23823  3.1  2.3 609508 92416 ?        Rl   11:06   0:01  \_ /usr/sbin/apache2 -k start
www-data 23827  2.4  1.8 468088 70168 ?        R    11:06   0:01  \_ /usr/sbin/apache2 -k start
www-data 23828  3.2  2.4 758192 93540 ?        Rl   11:06   0:01  \_ /usr/sbin/apache2 -k start
www-data 23829  2.4  2.0 685480 79588 ?        Sl   11:06   0:01  \_ /usr/sbin/apache2 -k start
www-data 23830  3.1  2.3 610384 92432 ?        Rl   11:06   0:01  \_ /usr/sbin/apache2 -k start
www-data 23832  1.9  1.7 683660 67220 ?        Sl   11:06   0:01  \_ /usr/sbin/apache2 -k start
www-data 23833  3.1  2.4 757420 94052 ?        Rl   11:06   0:01  \_ /usr/sbin/apache2 -k start
www-data 23835  2.7  1.7 746280 65712 ?        Rl   11:06   0:01  \_ /usr/sbin/apache2 -k start
www-data 23843  2.8  2.3 608176 90052 ?        Rl   11:06   0:01  \_ /usr/sbin/apache2 -k start
www-data 23844  2.6  1.9 683120 74080 ?        Sl   11:06   0:01  \_ /usr/sbin/apache2 -k start
www-data 23847  2.5  2.0 830664 80776 ?        Sl   11:06   0:01  \_ /usr/sbin/apache2 -k start
www-data 23848  2.6  2.3 609256 91384 ?        Sl   11:06   0:01  \_ /usr/sbin/apache2 -k start
www-data 23849  2.7  1.8 468188 71308 ?        R    11:06   0:01  \_ /usr/sbin/apache2 -k start
www-data 23850  2.8  1.8 684208 72724 ?        Rl   11:06   0:01  \_ /usr/sbin/apache2 -k start
www-data 23854  2.7  1.7 470312 68732 ?        R    11:06   0:01  \_ /usr/sbin/apache2 -k start
www-data 23855  2.5  1.9 609456 74208 ?        Rl   11:06   0:01  \_ /usr/sbin/apache2 -k start
www-data 23856  2.6  1.8 608136 71364 ?        Rl   11:06   0:01  \_ /usr/sbin/apache2 -k start
www-data 23857  2.7  1.8 470488 70996 ?        R    11:06   0:01  \_ /usr/sbin/apache2 -k start
www-data 23858  2.5  1.9 612324 76488 ?        Sl   11:06   0:01  \_ /usr/sbin/apache2 -k start
www-data 23860  2.5  1.9 681868 75532 ?        Sl   11:06   0:01  \_ /usr/sbin/apache2 -k start
www-data 23862  2.8  2.1 756112 83120 ?        Rl   11:06   0:01  \_ /usr/sbin/apache2 -k start
www-data 23864  2.1  1.8 608148 71348 ?        Sl   11:06   0:00  \_ /usr/sbin/apache2 -k start
www-data 23870  2.4  1.8 469780 69868 ?        R    11:06   0:01  \_ /usr/sbin/apache2 -k start
www-data 23872  2.4  2.2 538288 85420 ?        Sl   11:06   0:01  \_ /usr/sbin/apache2 -k start
www-data 23873  2.7  1.8 472252 71352 ?        R    11:06   0:01  \_ /usr/sbin/apache2 -k start
www-data 23874  4.0  2.3 610428 88744 ?        Sl   11:06   0:01  \_ /usr/sbin/apache2 -k start
www-data 23875  2.5  1.8 683228 70612 ?        Rl   11:06   0:01  \_ /usr/sbin/apache2 -k start
www-data 23876  2.5  2.0 834520 77692 ?        Rl   11:06   0:01  \_ /usr/sbin/apache2 -k start
www-data 23882  1.9  1.6 464340 62248 ?        R    11:06   0:00  \_ /usr/sbin/apache2 -k start
www-data 23885  3.2  1.8 472252 71408 ?        R    11:06   0:01  \_ /usr/sbin/apache2 -k start
www-data 23889  2.6  1.8 608416 71892 ?        Rl   11:06   0:01  \_ /usr/sbin/apache2 -k start
www-data 23891  2.6  1.7 470028 69152 ?        R    11:06   0:01  \_ /usr/sbin/apache2 -k start
www-data 23894  4.7  2.0 832144 78816 ?        Sl   11:06   0:01  \_ /usr/sbin/apache2 -k start
www-data 23895  1.7  1.7 468172 68940 ?        R    11:06   0:00  \_ /usr/sbin/apache2 -k start
www-data 23896  2.6  1.8 471232 71644 ?        S    11:06   0:01  \_ /usr/sbin/apache2 -k start
www-data 23897  1.2  1.3 664652 50848 ?        Sl   11:06   0:00  \_ /usr/sbin/apache2 -k start
www-data 23908  2.5  1.9 609456 74644 ?        Sl   11:06   0:00  \_ /usr/sbin/apache2 -k start
www-data 23911  2.2  1.6 462552 62668 ?        R    11:06   0:00  \_ /usr/sbin/apache2 -k start
www-data 23912  2.8  2.3 537456 90296 ?        Sl   11:06   0:01  \_ /usr/sbin/apache2 -k start
www-data 23918  0.3  1.0 452388 41720 ?        R    11:06   0:00  \_ /usr/sbin/apache2 -k start
www-data 23920  2.3  1.8 470140 72120 ?        R    11:06   0:00  \_ /usr/sbin/apache2 -k start
www-data 23922  1.7  1.8 607312 70044 ?        Rl   11:06   0:00  \_ /usr/sbin/apache2 -k start
www-data 23923  2.9  1.8 602388 69756 ?        Rl   11:06   0:00  \_ /usr/sbin/apache2 -k start
root     15803  0.0  0.0  36156  1412 ?        Ss   oct27   0:05 /usr/lib/postfix/master
postfix  15807  0.0  0.0  38380  1688 ?        S    oct27   0:01  \_ qmgr -l -t fifo -u
postfix  20494  0.0  0.0  42448  1160 ?        S    oct27   0:00  \_ tlsmgr -l -t unix -u -c
postfix  23635  0.0  0.0  38220  1132 ?        S    11:04   0:00  \_ anvil -l -t unix -u -c
postfix  23777  0.0  0.0  38220  1148 ?        S    11:06   0:00  \_ pickup -l -t fifo -u -c
postfix  23982  0.0  0.0  38232  2344 ?        S    11:07   0:00  \_ trivial-rewrite -n rewrite -t unix -u -c
postfix  23984  0.0  0.0  42548  2672 ?        S    11:07   0:00  \_ smtp -t unix -u -c
postfix  23985  0.0  0.0  42548  2672 ?        S    11:07   0:00  \_ smtp -t unix -u -c
postfix  23988  0.0  0.0  38256  2356 ?        S    11:07   0:00  \_ bounce -z -n defer -t unix -u -c
root      9452  0.0  0.1  17772  5500 ?        S<L  00:00   0:01 /usr/bin/atop -a -w /var/log/atop/atop_20171029 600
root     17826  0.0  0.0  52704  1188 ?        Ss   08:41   0:00 /usr/sbin/sshd
root     23980  0.1  0.0  71956  3684 ?        Ss   11:07   0:00  \_ sshd: root@pts/0 
root     23990  0.3  0.0  24332  3456 pts/0    Ss   11:07   0:00      \_ -bash
root     24085  0.0  0.0  14444  1452 pts/0    S+   11:07   0:00          \_ /bin/bash /var/scripts/diagnosis_instantanea.sh
root     24094  0.0  0.0  18408  1348 pts/0    R+   11:07   0:00              \_ ps auxwwwf

Thanks in advance.

  • Looks like it is apache doing it - does it track with the number of web requests made to your web server? I suspect you will get more information if you start monitoring your webservers stats over time and looking into the apache logs. – Michael Daffin Oct 29 '17 at 10:54
1

The simple first part of the answer is: You have a load of 39 because there are

Tasks: 148 total,  43 running,

43 processes on the run queue.

Of course, this leads to the next question: Are so many processes running because there are so many requests (check the logs) or because the handling of the requests is delayed.

There is no I/O which could be responsible. If mysqld or memcached was the bottleneck then we would se them among the processes with the highest CPU consumption.

Maybe your application (running within Apache) causes network requests and those cause the delay. Have a look at

netstat --inet --inet6 -nap | sort -k 6

in order to check whether there are processes with more than one connection.

Or, simpler (but with less information):

netstat --inet --inet6 -nap | sort -k 6 | awk '{ print $7 }' | uniq -cd
  • I'll add those commands to the log when it happens again. Right now, the load is 0.37, so the output wouldn't be relevant. Thank you. – Uxio Oct 29 '17 at 11:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.