0

I have an EC2 t2.micro (Linux AMI) instance. Since a few days I noticed that RAM is being 99% used by "ps" .

When that happens :

$ free -m
              total        used        free      shared  buff/cache   available
Mem:            983         561          79           0         342         257
Swap:             0           0           0

top shows

$ top
top - 05:58:52 up 17:28,  2 users,  load average: 1.00, 0.78, 0.39
Tasks:  95 total,   1 running,  58 sleeping,   0 stopped,   0 zombie
%Cpu(s):100.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem :  1006940 total,    80816 free,   575132 used,   350992 buff/cache
KiB Swap:        0 total,        0 free,        0 used.   263376 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
14386 ec2-user  20   0  307096 269360   4192 S 99.6 26.8   7:15.88 ps
 9019 ec2-user  20   0  771132 128884  47792 S  0.4 12.8   1:18.75 python
    1 root      20   0   45624   5372   3936 S  0.0  0.5   0:07.01 systemd
    2 root      20   0       0      0      0 S  0.0  0.0   0:00.05 kthreadd
    4 root       0 -20       0      0      0 I  0.0  0.0   0:00.00 kworker/0:0H
    6 root       0 -20       0      0      0 I  0.0  0.0   0:00.00 mm_percpu_wq
    7 root      20   0       0      0      0 S  0.0  0.0   0:00.52 ksoftirqd/0
    8 root      20   0       0      0      0 I  0.0  0.0   0:03.64 rcu_sched
    9 root      20   0       0      0      0 I  0.0  0.0   0:00.00 rcu_bh
   10 root      rt   0       0      0      0 S  0.0  0.0   0:00.00 migration/0
   11 root      rt   0       0      0      0 S  0.0  0.0   0:00.15 watchdog/0
   12 root      20   0       0      0      0 S  0.0  0.0   0:00.00 cpuhp/0
   14 root      20   0       0      0      0 S  0.0  0.0   0:00.00 kdevtmpfs
   15 root       0 -20       0      0      0 I  0.0  0.0   0:00.00 netns
   16 root      20   0       0      0      0 I  0.0  0.0   0:00.33 kworker/u30:1
   21 root      20   0       0      0      0 S  0.0  0.0   0:00.00 xenbus
   22 root      20   0       0      0      0 S  0.0  0.0   0:00.06 xenwatch
  172 root      20   0       0      0      0 S  0.0  0.0   0:00.01 khungtaskd
  173 root      20   0       0      0      0 S  0.0  0.0   0:00.00 oom_reaper
  174 root       0 -20       0      0      0 I  0.0  0.0   0:00.00 writeback
  176 root      20   0       0      0      0 S  0.0  0.0   0:00.00 kcompactd0
  177 root      25   5       0      0      0 S  0.0  0.0   0:00.00 ksmd
  178 root      39  19       0      0      0 S  0.0  0.0   0:00.00 khugepaged

Once I kill the ps process the system comes to normal state .

$ free -m
              total        used        free      shared  buff/cache   available
Mem:            983         294         343           0         345         524
Swap:             0           0           0

[ec2-user@ip-10-0-0-86 scenAI]$ top
top - 06:05:36 up 17:35,  1 user,  load average: 0.00, 0.20, 0.25
Tasks:  88 total,   1 running,  51 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.3 us,  0.0 sy,  0.0 ni, 99.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem :  1006940 total,   351520 free,   301488 used,   353932 buff/cache
KiB Swap:        0 total,        0 free,        0 used.   537044 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
 9017 ec2-user  20   0  231592  21596   9392 S  0.3  2.1   1:32.00 uvicorn
 9019 ec2-user  20   0  771132 128884  47792 S  0.3 12.8   1:19.29 python
    1 root      20   0   45624   5372   3936 S  0.0  0.5   0:07.03 systemd
    2 root      20   0       0      0      0 S  0.0  0.0   0:00.05 kthreadd

Though I am not directly running "ps" command random calls seem to be invoking it and once the script fail, checking "top" shows "ps" running at 99% CPU .

Also previously ps -ef|grep used to work but now gives ps: unrecognized option: e ps: unrecognized option: f .

** Edit ** Even ps -ef|grep ec2-user hangs :

[root@xxxxxx ec2-user]# ps -ef|grep ec2-user
ec2-user  9017  0.1  1.8 231592 18276 ?        S    Feb22   1:41 /home/ec2-user/FastAPI/apienv/bin/python /home/ec2-user
/FastAPI/apienv/bin/uvicorn -.......
ec2-user  9018  0.0  0.9 157884  9124 ?        S    Feb22   0:00 /home/ec2-user/FastAPI/apienv/bin/python -c from multiprocessing.semaphore_tracker import main;main(4)
ec2-user  9019  0.1 10.9 771132 110460 ?       S    Feb22   1:27 /home/ec2-user/FastAPI/apienv/bin/python -c from multiprocessing.spawn import spawn_main; spawn_main(tracker_fd=5, pipe_handle=7) --multiprocessing-fork
root     13290  0.0  0.6 154688  6860 ?        Ss   05:26   0:00 sshd: ec2-user [priv]
ec2-user 13308  0.0  0.3 154824  3460 ?        S    05:26   0:00 sshd: ec2-user@pts/1
ec2-user 13309  0.0  0.6 129176  6320 pts/1    Ss   05:26   0:00 -bash
ec2-user 15102  0.0  7.4 444216 75372 ?        Sl   05:59   0:00 /usr/lib64/R/bin/exec/R --slave --no-restore --file=xxxxxxx
root     16096  0.0  0.0 123540   952 pts/1    S+   07:44   0:00 grep --color=auto ec2-user
ps: unrecognized option: e
ps: unrecognized option: f
^C

(Did ctrl+c to quit) At the time top shows :

# top
top - 07:50:05 up 19:19,  1 user,  load average: 1.00, 0.69, 0.32
Tasks:  93 total,   1 running,  55 sleeping,   0 stopped,   0 zombie
%Cpu(s): 99.7 us,  0.3 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem :  1006940 total,   258552 free,   532292 used,   216096 buff/cache
KiB Swap:        0 total,        0 free,        0 used.   327024 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
16102 root      20   0  307088   4932   3992 S 99.9  0.5   5:44.08 ps
 9019 ec2-user  20   0  771132 110460  29368 S  0.3 11.0   1:27.82 python
    1 root      20   0   45624   4516   3080 S  0.0  0.4   0:07.09 systemd

Appreciate any pointers.

regards, Thomas

7
  • Please run ps uf $(ps -o ppid= -p 14386) to get information on the parent process. (14386 to be replaced by the pid of your ps process) (wich process is launching that strange ps process)
    – pLumo
    Feb 23, 2021 at 7:34
  • Since long text could not be added I edited the post . Even ps -ef|grep hangs using 99% CPU. Could ps have got corrupted ? Is it safe to remove and install procps again ? Feb 23, 2021 at 7:55
  • The original ps -ef command is working -- it produces output in the required format for -ef which gets properly grepped. Then it runs something else called ps which is not /bin/ps. You might check such things as your aliases and command -V ps. I think you have another ps command somewhere that attempts to loop in /bin/ps but fails after the first iteration. Feb 23, 2021 at 9:32
  • [ec2-user@xxx ~]$ command -V ps ps is /usr/bin/ps . Feb 23, 2021 at 10:17
  • Is it possible to reinstall procps. Will it put back a good ps in the /usr/bin location. If I do yum install it says requirement already satisfied. Feb 23, 2021 at 10:32

1 Answer 1

1

This might be a crypto miner and you've been hacked. You do not recover, you're now 100% compromised and you cannot trust this system. Save the data, wipe the instance, install from scratch and properly secure your system.

For more info consult with:

https://security.stackexchange.com/questions/39231/how-do-i-deal-with-a-compromised-server

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.