0

I have an EC2 t2.micro (Linux AMI) instance. Since a few days I noticed that RAM is being 99% used by "ps" .

When that happens :

$ free -m
              total        used        free      shared  buff/cache   available
Mem:            983         561          79           0         342         257
Swap:             0           0           0

top shows

$ top
top - 05:58:52 up 17:28,  2 users,  load average: 1.00, 0.78, 0.39
Tasks:  95 total,   1 running,  58 sleeping,   0 stopped,   0 zombie
%Cpu(s):100.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem :  1006940 total,    80816 free,   575132 used,   350992 buff/cache
KiB Swap:        0 total,        0 free,        0 used.   263376 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
14386 ec2-user  20   0  307096 269360   4192 S 99.6 26.8   7:15.88 ps
 9019 ec2-user  20   0  771132 128884  47792 S  0.4 12.8   1:18.75 python
    1 root      20   0   45624   5372   3936 S  0.0  0.5   0:07.01 systemd
    2 root      20   0       0      0      0 S  0.0  0.0   0:00.05 kthreadd
    4 root       0 -20       0      0      0 I  0.0  0.0   0:00.00 kworker/0:0H
    6 root       0 -20       0      0      0 I  0.0  0.0   0:00.00 mm_percpu_wq
    7 root      20   0       0      0      0 S  0.0  0.0   0:00.52 ksoftirqd/0
    8 root      20   0       0      0      0 I  0.0  0.0   0:03.64 rcu_sched
    9 root      20   0       0      0      0 I  0.0  0.0   0:00.00 rcu_bh
   10 root      rt   0       0      0      0 S  0.0  0.0   0:00.00 migration/0
   11 root      rt   0       0      0      0 S  0.0  0.0   0:00.15 watchdog/0
   12 root      20   0       0      0      0 S  0.0  0.0   0:00.00 cpuhp/0
   14 root      20   0       0      0      0 S  0.0  0.0   0:00.00 kdevtmpfs
   15 root       0 -20       0      0      0 I  0.0  0.0   0:00.00 netns
   16 root      20   0       0      0      0 I  0.0  0.0   0:00.33 kworker/u30:1
   21 root      20   0       0      0      0 S  0.0  0.0   0:00.00 xenbus
   22 root      20   0       0      0      0 S  0.0  0.0   0:00.06 xenwatch
  172 root      20   0       0      0      0 S  0.0  0.0   0:00.01 khungtaskd
  173 root      20   0       0      0      0 S  0.0  0.0   0:00.00 oom_reaper
  174 root       0 -20       0      0      0 I  0.0  0.0   0:00.00 writeback
  176 root      20   0       0      0      0 S  0.0  0.0   0:00.00 kcompactd0
  177 root      25   5       0      0      0 S  0.0  0.0   0:00.00 ksmd
  178 root      39  19       0      0      0 S  0.0  0.0   0:00.00 khugepaged

Once I kill the ps process the system comes to normal state .

$ free -m
              total        used        free      shared  buff/cache   available
Mem:            983         294         343           0         345         524
Swap:             0           0           0

[ec2-user@ip-10-0-0-86 scenAI]$ top
top - 06:05:36 up 17:35,  1 user,  load average: 0.00, 0.20, 0.25
Tasks:  88 total,   1 running,  51 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.3 us,  0.0 sy,  0.0 ni, 99.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem :  1006940 total,   351520 free,   301488 used,   353932 buff/cache
KiB Swap:        0 total,        0 free,        0 used.   537044 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
 9017 ec2-user  20   0  231592  21596   9392 S  0.3  2.1   1:32.00 uvicorn
 9019 ec2-user  20   0  771132 128884  47792 S  0.3 12.8   1:19.29 python
    1 root      20   0   45624   5372   3936 S  0.0  0.5   0:07.03 systemd
    2 root      20   0       0      0      0 S  0.0  0.0   0:00.05 kthreadd

Though I am not directly running "ps" command random calls seem to be invoking it and once the script fail, checking "top" shows "ps" running at 99% CPU .

Also previously ps -ef|grep used to work but now gives ps: unrecognized option: e ps: unrecognized option: f .

** Edit ** Even ps -ef|grep ec2-user hangs :

[root@xxxxxx ec2-user]# ps -ef|grep ec2-user
ec2-user  9017  0.1  1.8 231592 18276 ?        S    Feb22   1:41 /home/ec2-user/FastAPI/apienv/bin/python /home/ec2-user
/FastAPI/apienv/bin/uvicorn -.......
ec2-user  9018  0.0  0.9 157884  9124 ?        S    Feb22   0:00 /home/ec2-user/FastAPI/apienv/bin/python -c from multiprocessing.semaphore_tracker import main;main(4)
ec2-user  9019  0.1 10.9 771132 110460 ?       S    Feb22   1:27 /home/ec2-user/FastAPI/apienv/bin/python -c from multiprocessing.spawn import spawn_main; spawn_main(tracker_fd=5, pipe_handle=7) --multiprocessing-fork
root     13290  0.0  0.6 154688  6860 ?        Ss   05:26   0:00 sshd: ec2-user [priv]
ec2-user 13308  0.0  0.3 154824  3460 ?        S    05:26   0:00 sshd: ec2-user@pts/1
ec2-user 13309  0.0  0.6 129176  6320 pts/1    Ss   05:26   0:00 -bash
ec2-user 15102  0.0  7.4 444216 75372 ?        Sl   05:59   0:00 /usr/lib64/R/bin/exec/R --slave --no-restore --file=xxxxxxx
root     16096  0.0  0.0 123540   952 pts/1    S+   07:44   0:00 grep --color=auto ec2-user
ps: unrecognized option: e
ps: unrecognized option: f
^C

(Did ctrl+c to quit) At the time top shows :

# top
top - 07:50:05 up 19:19,  1 user,  load average: 1.00, 0.69, 0.32
Tasks:  93 total,   1 running,  55 sleeping,   0 stopped,   0 zombie
%Cpu(s): 99.7 us,  0.3 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem :  1006940 total,   258552 free,   532292 used,   216096 buff/cache
KiB Swap:        0 total,        0 free,        0 used.   327024 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
16102 root      20   0  307088   4932   3992 S 99.9  0.5   5:44.08 ps
 9019 ec2-user  20   0  771132 110460  29368 S  0.3 11.0   1:27.82 python
    1 root      20   0   45624   4516   3080 S  0.0  0.4   0:07.09 systemd

Appreciate any pointers.

regards, Thomas

7
  • Please run ps uf $(ps -o ppid= -p 14386) to get information on the parent process. (14386 to be replaced by the pid of your ps process) (wich process is launching that strange ps process)
    – pLumo
    Feb 23, 2021 at 7:34
  • Since long text could not be added I edited the post . Even ps -ef|grep hangs using 99% CPU. Could ps have got corrupted ? Is it safe to remove and install procps again ? Feb 23, 2021 at 7:55
  • The original ps -ef command is working -- it produces output in the required format for -ef which gets properly grepped. Then it runs something else called ps which is not /bin/ps. You might check such things as your aliases and command -V ps. I think you have another ps command somewhere that attempts to loop in /bin/ps but fails after the first iteration. Feb 23, 2021 at 9:32
  • [ec2-user@xxx ~]$ command -V ps ps is /usr/bin/ps . Feb 23, 2021 at 10:17
  • Is it possible to reinstall procps. Will it put back a good ps in the /usr/bin location. If I do yum install it says requirement already satisfied. Feb 23, 2021 at 10:32

1 Answer 1

1

This might be a crypto miner and you've been hacked. You do not recover, you're now 100% compromised and you cannot trust this system. Save the data, wipe the instance, install from scratch and properly secure your system.

For more info consult with:

https://security.stackexchange.com/questions/39231/how-do-i-deal-with-a-compromised-server

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .