-1

Some time ago our site had been hacked, and we were able to recover successfully, however there still seems to be something weird that we havent got rid off yet. Once per hour, there is a job starting that is consuming a lot of CPU resources. CPU usage

I have installed a job running in background which is doing

ps aux u

whenever sys_getloadavg() gets higher than 0.4 When I look at the output, I am puzzled, there's almost no visible CPU activity although sys_getloadavg() showed a value of 3.34 in this example. The server response is indeed extremely when this happens. How can this be explained?

USER      PID %CPU %MEM    VSZ  RSS TTY      STAT START  TIME COMMAND
root        1  0.0  0.4 225468  4356 ?        Ss  Nov07  0:29 /sbin/init
root        2  0.0  0.0      0    0 ?        S    Nov07  0:00 [kthreadd]
root        4  0.0  0.0      0    0 ?        I<  Nov07  0:00 [kworker/0:0H]
root        6  0.0  0.0      0    0 ?        I<  Nov07  0:00 [mm_percpu_wq]
root        7  0.0  0.0      0    0 ?        S    Nov07  1:14 [ksoftirqd/0]
root        8  0.0  0.0      0    0 ?        I    Nov07  2:01 [rcu_sched]
root        9  0.0  0.0      0    0 ?        I    Nov07  0:00 [rcu_bh]
root        10  0.0  0.0      0    0 ?        S    Nov07  0:00 [migration/0]
root        11  0.0  0.0      0    0 ?        S    Nov07  0:04 [watchdog/0]
root        12  0.0  0.0      0    0 ?        S    Nov07  0:00 [cpuhp/0]
root        13  0.0  0.0      0    0 ?        S    Nov07  0:00 [kdevtmpfs]
root        14  0.0  0.0      0    0 ?        I<  Nov07  0:00 [netns]
root        15  0.0  0.0      0    0 ?        S    Nov07  0:00 [rcu_tasks_kthre]
root        16  0.0  0.0      0    0 ?        S    Nov07  0:03 [kauditd]
root        17  0.0  0.0      0    0 ?        S    Nov07  0:00 [xenbus]
root        18  0.0  0.0      0    0 ?        S    Nov07  0:00 [xenwatch]
root        20  0.0  0.0      0    0 ?        S    Nov07  0:00 [khungtaskd]
root        21  0.0  0.0      0    0 ?        S    Nov07  0:05 [oom_reaper]
root        22  0.0  0.0      0    0 ?        I<  Nov07  0:00 [writeback]
root        23  0.0  0.0      0    0 ?        S    Nov07  0:00 [kcompactd0]
root        24  0.0  0.0      0    0 ?        SN  Nov07  0:00 [ksmd]
root        25  0.0  0.0      0    0 ?        SN  Nov07  6:07 [khugepaged]
root        26  0.0  0.0      0    0 ?        I<  Nov07  0:00 [crypto]
root        27  0.0  0.0      0    0 ?        I<  Nov07  0:00 [kintegrityd]
root        28  0.0  0.0      0    0 ?        I<  Nov07  0:00 [kblockd]
root        29  0.0  0.0      0    0 ?        I<  Nov07  0:00 [ata_sff]
root        30  0.0  0.0      0    0 ?        I<  Nov07  0:00 [md]
root        31  0.0  0.0      0    0 ?        I<  Nov07  0:00 [edac-poller]
root        32  0.0  0.0      0    0 ?        I<  Nov07  0:00 [devfreq_wq]
root        33  0.0  0.0      0    0 ?        I<  Nov07  0:00 [watchdogd]
root        36  0.0  0.0      0    0 ?        S    Nov07  1:51 [kswapd0]
root        37  0.0  0.0      0    0 ?        I<  Nov07  0:00 [kworker/u31:0]
root        38  0.0  0.0      0    0 ?        S    Nov07  0:00 [ecryptfs-kthrea]
root        80  0.0  0.0      0    0 ?        I<  Nov07  0:00 [kthrotld]
root        81  0.0  0.0      0    0 ?        I<  Nov07  0:00 [acpi_thermal_pm]
root        82  0.0  0.0      0    0 ?        S    Nov07  0:00 [scsi_eh_0]
root        83  0.0  0.0      0    0 ?        I<  Nov07  0:00 [scsi_tmf_0]
root        84  0.0  0.0      0    0 ?        S    Nov07  0:00 [scsi_eh_1]
root        85  0.0  0.0      0    0 ?        I<  Nov07  0:00 [scsi_tmf_1]
root        91  0.0  0.0      0    0 ?        I<  Nov07  0:00 [ipv6_addrconf]
root      100  0.0  0.0      0    0 ?        I<  Nov07  0:00 [kstrp]
root      101  0.0  0.0      0    0 ?        I<  Nov07  6:08 [kworker/0:1H]
root      118  0.0  0.0      0    0 ?        I<  Nov07  0:00 [charger_manager]
root      182  0.0  0.0      0    0 ?        I<  Nov07  0:00 [ttm_swap]
root      265  0.0  0.0      0    0 ?        I<  Nov07  0:00 [raid5wq]
root      317  0.0  0.0      0    0 ?        S    Nov07  6:36 [jbd2/xvda1-8]
root      318  0.0  0.0      0    0 ?        I<  Nov07  0:00 [ext4-rsv-conver]
root      401  0.0  0.0      0    0 ?        I<  Nov07  0:00 [iscsi_eh]
root      412  0.0  0.0      0    0 ?        I<  Nov07  0:00 [ib-comp-wq]
root      413  0.0  0.0      0    0 ?        I<  Nov07  0:00 [ib_mcast]
root      414  0.0  0.0      0    0 ?        I<  Nov07  0:00 [ib_nl_sa_wq]
root      420  0.0  0.0      0    0 ?        I<  Nov07  0:00 [rdma_cm]
root      421  0.0  3.0 421780 30768 ?        S<s  Nov07  2:57 /lib/systemd/systemd-journald
root      435  0.0  0.0 105900  840 ?        Ss  Nov07  0:00 /sbin/lvmetad -f
root      443  0.0  0.3  45064  3152 ?        Ss  Nov07  1:23 /lib/systemd/systemd-udevd
systemd+  574  0.0  0.2 141924  2812 ?        Ssl  Nov07  0:01 /lib/systemd/systemd-timesyncd
root      586  0.0  0.0  31976  264 ?        S<sl Nov07  0:10 /sbin/auditd
root      822  0.0  0.2  26016  2604 ?        Ss  Nov07  0:01 /sbin/dhclient -1 -4 -v -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases -I -df /var/lib/dhcp/dhclient6.eth0.leases eth0
clamav    1017  0.0  0.7 152884  7884 ?        Ss  Nov07  11:36 /usr/bin/freshclam -d --foreground=true
root      1019  0.0  0.1  4520  1600 ?        S<  Nov07  0:23 /usr/sbin/atopacctd
root      1039  0.0  0.1 286464  1396 ?        Ssl  Nov07  0:39 /usr/lib/accountsservice/accounts-daemon
root      1045  0.0  0.2  30024  2396 ?        Ss  Nov07  0:50 /usr/sbin/cron -f
syslog    1049  0.0  0.1 267268  1892 ?        Ssl  Nov07  0:25 /usr/sbin/rsyslogd -n
root      1050  0.0  0.1 636120  1448 ?        Ssl  Nov07  0:04 /usr/bin/lxcfs /var/lib/lxcfs/
message+  1060  0.0  0.2  50384  2940 ?        Ss  Nov07  0:06 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
root      1077  0.0  0.1  70640  1576 ?        Ss  Nov07  0:03 /lib/systemd/systemd-logind
daemon    1082  0.0  0.1  28328  1572 ?        Ss  Nov07  0:00 /usr/sbin/atd -f
root      1093  0.0  1.0 169092 10204 ?        Ssl  Nov07  0:00 /usr/bin/python3 /usr/bin/networkd-dispatcher --run-startup-triggers
root      1103  0.0  0.1 288876  1048 ?        Ssl  Nov07  0:02 /usr/lib/policykit-1/polkitd --no-debug
root      1123  0.0  0.1  14664  1760 ttyS0    Ss+  Nov07  0:00 /sbin/agetty -o -p -- \u --keep-baud 115200,38400,9600 ttyS0 vt220
root      1128  0.0  0.1  14888  1540 tty1    Ss+  Nov07  0:00 /sbin/agetty -o -p -- \u --noclear tty1 linux
root      1161  0.0  1.0 185936 10328 ?        Ssl  Nov07  0:00 /usr/bin/python3 /usr/share/unattended-upgrades/unattended-upgrade-shutdown --wait-for-signal
www-data  1202  0.0  0.0  19904  360 ?        Ss  Nov07  0:37 /usr/bin/htcacheclean -d 120 -p /var/cache/apache2/mod_cache_disk -l 300M -n
root      1386  0.0  2.6  95296 26772 ?        Ss  Nov07  0:27 /usr/bin/perl /usr/share/webmin/miniserv.pl /etc/webmin/miniserv.conf
root      1465  0.0  0.2  67372  2424 ?        Ss  Nov07  0:23 /usr/lib/postfix/sbin/master -w
postfix  1482  0.0  0.2  73956  2408 ?        S    Nov07  0:51 qmgr -l -t unix -u
root      2067  0.0  3.2 485040 32928 ?        Ss  Nov08  7:46 /usr/sbin/apache2 -k start
root    12896  0.0  0.2  72292  2316 ?        Ss  Nov25  0:00 /usr/sbin/sshd -D
mysql    16706  1.1 30.5 1160824 308172 ?      Sl  16:18  2:11 /usr/sbin/mysqld --daemonize --pid-file=/run/mysqld/mysqld.pid
root    17345  0.0  0.2  57504  2496 ?        S    16:30  0:00 /usr/sbin/CRON -f
ubuntu  17347  0.0  1.4 291368 14852 ?        Ss  16:30  0:00 /usr/bin/php /var/www/html/cron_schedule_ride_new.php
root    19490  0.0  0.9  26416  9304 ?        S<Ls 00:00  0:18 /usr/bin/atop -R -w /var/log/atop/atop_20191127 600
root    20107  0.0  0.0      0    0 ?        I    17:56  0:00 [kworker/0:0]
postfix  20251  0.0  0.2  73808  2292 ?        S    18:08  0:00 pickup -l -t unix -u -c
www-data 20898  0.1  5.0 564828 51008 ?        S    18:36  0:05 /usr/sbin/apache2 -k start
www-data 21260  0.1  2.7 563672 27968 ?        S    18:47  0:02 /usr/sbin/apache2 -k start
www-data 21262  0.1  5.0 564388 51360 ?        S    18:47  0:03 /usr/sbin/apache2 -k start
root    21534  0.0  0.0      0    0 ?        I    19:05  0:00 [kworker/u30:1]
systemd+ 21845  0.0  0.3  70624  3364 ?        Ss  19:12  0:00 /lib/systemd/systemd-resolved
root    21864  0.0  0.0      0    0 ?        I    19:12  0:00 [kworker/u30:2]
root    21899  0.0  0.0      0    0 ?        I    19:12  0:00 [kworker/0:1]
www-data 21943  0.0  2.4 488332 24828 ?        S    19:15  0:00 /usr/sbin/apache2 -k start
root    21981  0.0  0.0      0    0 ?        I    19:18  0:00 [kworker/u30:0]
www-data 21985  0.1  2.4 488424 24532 ?        S    19:18  0:00 /usr/sbin/apache2 -k start
www-data 22004  0.1  2.3 487908 23308 ?        S    19:19  0:00 /usr/sbin/apache2 -k start
www-data 22052  0.0  2.9 487908 29296 ?        S    19:21  0:00 /usr/sbin/apache2 -k start
www-data 22053  0.0  0.0      0    0 ?        Z    19:21  0:00 [apache2] <defunct>
www-data 22057  0.1  3.0 490388 30400 ?        S    19:21  0:00 /usr/sbin/apache2 -k start
www-data 22058  0.0  2.6 487776 26228 ?        S    19:21  0:00 /usr/sbin/apache2 -k start
www-data 22059  0.1  2.9 488332 29684 ?        S    19:21  0:00 /usr/sbin/apache2 -k start
www-data 22060  0.0  2.9 487908 29312 ?        S    19:21  0:00 /usr/sbin/apache2 -k start
www-data 22067  0.0  2.9 488332 29432 ?        S    19:21  0:00 /usr/sbin/apache2 -k start
www-data 22069  0.0  2.9 487908 29436 ?        S    19:21  0:00 /usr/sbin/apache2 -k start
www-data 22070  0.0  2.9 488332 29556 ?        S    19:21  0:00 /usr/sbin/apache2 -k start
www-data 22077  0.0  2.9 487908 29300 ?        S    19:21  0:00 /usr/sbin/apache2 -k start
www-data 22078  0.0  2.6 487776 26228 ?        S    19:21  0:00 /usr/sbin/apache2 -k start
www-data 22085  0.0  2.8 487908 29112 ?        S    19:21  0:00 /usr/sbin/apache2 -k start
www-data 22086  0.0  2.9 488332 29452 ?        S    19:21  0:00 /usr/sbin/apache2 -k start
www-data 22088  0.0  2.8 487908 29172 ?        S    19:21  0:00 /usr/sbin/apache2 -k start
www-data 22090  0.0  2.9 488304 29848 ?        S    19:21  0:00 /usr/sbin/apache2 -k start
www-data 22093  0.0  2.7 487776 27908 ?        S    19:21  0:00 /usr/sbin/apache2 -k start
www-data 22094  0.0  2.9 488332 29668 ?        S    19:21  0:00 /usr/sbin/apache2 -k start
www-data 22099  0.0  3.0 488388 30292 ?        S    19:21  0:00 /usr/sbin/apache2 -k start
root    22112  0.0  0.3  57504  3360 ?        S    19:22  0:00 /usr/sbin/CRON -f
root    22113  0.0  0.3  57504  3356 ?        S    19:22  0:00 /usr/sbin/CRON -f
root    22114  0.0  0.3  11592  3248 ?        Ss  19:22  0:00 /bin/bash -c /usr/bin/php /var/www/html/top_log.php >/dev/null 2>&1
www-data 22115  0.0  0.0  4628  856 ?        Ss  19:22  0:00 /bin/sh -c /usr/bin/php /var/www/html/crondeleteico.php >/dev/null 2>&1
www-data 22116 10.0  2.4 284932 24440 ?        S    19:22  0:00 /usr/bin/php /var/www/html/crondeleteico.php
root    22117  8.0  2.2 284932 22464 ?        SN  19:22  0:00 /usr/bin/php /var/www/html/top_log.php
root    22120  0.0  0.0  4628  808 ?        SN  19:22  0:00 sh -c ps aux u
root    22121  0.0  0.3  36072  3376 ?        RN  19:22  0:00 ps aux u
root    23492  0.0  2.4  95296 25032 ?        S    Nov11  0:00 /usr/bin/perl /usr/share/webmin/miniserv.pl /etc/webmin/miniserv.conf
  • Related and definitly worth a read: security.stackexchange.com/questions/39231/… – Panki Nov 28 '19 at 8:38
  • read it several times, very useful, helped a lot to clean stuff from server. But there's still one thing (this one) I cant get rid of – michaelsmith Nov 28 '19 at 8:40
  • If possible, trash the server and start from scratch (reinstall). – Panki Nov 28 '19 at 8:41
  • This is the very last resort. Site structure is complex and we have 15 millions db connections per month and would like to find a way to avoid server downtime as much as possible – michaelsmith Nov 28 '19 at 8:43
  • Does any user (including root) have cron jobs running around the times of the peaks? – Kusalananda Nov 28 '19 at 9:12
0

Hackers often replace tools like top, ps and the like with 'versions' that do not show the hacker's traces and processes.

Try to use different tools to find the rogue processes. And obviously try to re-install the ps, top and other utilities.

| improve this answer | |
  • 1
    Shared libraries may also be replaced – wurtel Nov 28 '19 at 8:51
  • I have done apt-get install --reinstall procps now, will see how that will work out. But would rootkit scanners not have to find this if that was the case? – michaelsmith Nov 28 '19 at 9:13
  • Yes, that's mainly what rootkit scanner are supposed to do. But they can be dodged or replaced too. – Hkoof Nov 28 '19 at 9:26
  • any suggestion which rootkit malware scanner would find something like this? – michaelsmith Nov 28 '19 at 9:29
  • checksum the binaries and libraries against known good ones of the same version. You could use md5sum or shasum (or the many other *sum variants). Debian also has debsums program which helps if they haven't fiddled with the checksum database. – Craig Small Apr 2 at 3:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.