suspicious activity in the server?

Question

I have a remote server and I got email about 90% consumption of bandwidth usage. So I logged in the server and checked and noticed that gnome-terminal was running in the server, which I haven't installed. I got it from the netstat output.

    [root@ser ~]# netstat -tunlpa
    Active Internet connections (servers and established)
    Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
    tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      118/sshd            
    tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      223/sendmail: accep 
    tcp        0     28 xx.xx.xx.xx:22       MYIP:42584     ESTABLISHED 750/sshd: givi [p 
    tcp        0      0 xx.xx.xx.xx:55920    204.44.101.130:21       ESTABLISHED 379/gnome-terminal  
    tcp6       0      0 :::22                   :::*                    LISTEN      118/sshd            

This is a new server and I haven't started any apps except, sendmail and sshd(already ruuning), but I have installed few mandatory applications. In the ps result also I am seeing some activity(like commands are running) which I can't figure out how to solve?

[root@ser ~]# ps auxxwwffff
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.3  40892  3268 ?        Ss   05:03   0:00 init -z       
root         2  0.0  0.0      0     0 ?        S    05:03   0:00 [kthreadd/5227]
root         3  0.0  0.0      0     0 ?        S    05:03   0:00  \_ [khelper/5227]
root        60  0.0  0.1  41452  1684 ?        Ss   05:03   0:00 /usr/lib/systemd/systemd-udevd
root        67  0.0  0.4 148056  5144 ?        Ss   05:03   0:00 /usr/lib/systemd/systemd-journald
dbus        97  0.0  0.1  26400  1604 ?        Ss   05:03   0:00 /bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
root       104  0.0  0.1  26332  1628 ?        Ss   05:03   0:00 /usr/lib/systemd/systemd-logind
root       118  0.0  0.3  82788  3556 ?        Ss   05:03   0:00 /usr/sbin/sshd -D
root       750  0.0  0.4 132876  5080 ?        Ss   05:05   0:00  \_ sshd: givi [priv]
givinv     752  0.0  0.2 132876  2236 ?        S    05:05   0:00      \_ sshd: givi@pts/0
givi     753  0.0  0.1 115336  2024 pts/0    Ss   05:05   0:00          \_ -bash
root       790  0.0  0.2 187644  2744 pts/0    S    05:05   0:00              \_ sudo -i
root       791  0.0  0.1 115336  2064 pts/0    S    05:05   0:00                  \_ -bash
root      2276  0.0  0.1 139444  1600 pts/0    R+   05:12   0:00                      \_ ps auxxwwffff
root       127  0.0  0.0   6400   804 tty1     Ss+  05:03   0:00 /sbin/agetty --noclear --keep-baud console 115200 38400 9600 vt220
root       128  0.0  0.0   6400   804 tty2     Ss+  05:03   0:00 /sbin/agetty --noclear tty2 linux
root       129  0.0  0.1  22772  1560 ?        Ss   05:03   0:00 /usr/sbin/crond -n
root      1126  0.0  0.0  31520   300 ?        Ssl  05:07   0:00 sh
root      2248  0.0  0.0   1408   932 ?        Ss   05:12   0:00 cat resolv.conf
root      2251  0.0  0.0   1408   928 ?        Ss   05:12   0:00 cd /etc
root      2254  0.0  0.0   1408   932 ?        Ss   05:12   0:00 sleep 1
root      2256  0.0  0.0   1408   932 ?        Ss   05:12   0:00 ls
root      2257  0.0  0.0   1408   932 ?        Ss   05:12   0:00 netstat -antop
root      2266  0.0  0.0   1408   932 ?        Ss   05:12   0:00 uptime
root      2267  0.0  0.0   1408   932 ?        Ss   05:12   0:00 whoami
root      2272  0.0  0.0   1408   928 ?        Ss   05:12   0:00 who
root      2274  0.0  0.0   1408   932 ?        Ss   05:12   0:00 id
root      2275  0.0  0.0   1408   932 ?        Ss   05:12   0:00 sleep 1

See the last few lines of the result! If I run ps again, I will get different result for last few lines.

[root@ser ~]# ps auxxwwffff
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.3  40892  3268 ?        Ss   05:03   0:00 init -z       
root         2  0.0  0.0      0     0 ?        S    05:03   0:00 [kthreadd/5227]
root         3  0.0  0.0      0     0 ?        S    05:03   0:00  \_ [khelper/5227]
root        60  0.0  0.1  41452  1684 ?        Ss   05:03   0:00 /usr/lib/systemd/systemd-udevd
root        67  0.0  0.4 148056  5156 ?        Ss   05:03   0:00 /usr/lib/systemd/systemd-journald
dbus        97  0.0  0.1  26400  1604 ?        Ss   05:03   0:00 /bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
root       104  0.0  0.1  26332  1628 ?        Ss   05:03   0:00 /usr/lib/systemd/systemd-logind
root       118  0.0  0.3  82788  3556 ?        Ss   05:03   0:00 /usr/sbin/sshd -D
root       750  0.0  0.4 132876  5080 ?        Ss   05:05   0:00  \_ sshd: givi [priv]
givi     752  0.0  0.2 132876  2236 ?        S    05:05   0:00  |   \_ sshd: givi@pts/0
givi     753  0.0  0.1 115336  2024 pts/0    Ss   05:05   0:00  |       \_ -bash
root       790  0.0  0.2 187644  2744 pts/0    S    05:05   0:00  |           \_ sudo -i
root       791  0.0  0.1 115336  2064 pts/0    S    05:05   0:00  |               \_ -bash
root      2461  0.0  0.1 139444  1604 pts/0    R+   05:13   0:00  |                   \_ ps auxxwwffff
root      2459  0.0  0.3  84132  3636 ?        Ss   05:13   0:00  \_ sshd: [accepted]
sshd      2460  0.0  0.1  84132  1620 ?        S    05:13   0:00      \_ sshd: [net]
root       127  0.0  0.0   6400   804 tty1     Ss+  05:03   0:00 /sbin/agetty --noclear --keep-baud console 115200 38400 9600 vt220
root       128  0.0  0.0   6400   804 tty2     Ss+  05:03   0:00 /sbin/agetty --noclear tty2 linux
root       129  0.0  0.1  22772  1560 ?        Ss   05:03   0:00 /usr/sbin/crond -n
root      1126  0.0  0.0  31520   300 ?        Ssl  05:07   0:00 sh
root      2449  0.0  0.0   1408   932 ?        Ss   05:13   0:00 ls -la
root      2452  0.0  0.0   1408   936 ?        Ss   05:13   0:00 ls -la
root      2455  0.0  0.0   1408   928 ?        Ss   05:13   0:00 sleep 1
root      2457  0.0  0.0   1408   932 ?        Ss   05:13   0:00 who
root      2458  0.0  0.0   1408   932 ?        Ss   05:13   0:00 cat resolv.conf

I killed the PID I got but some sh process started afterwards(Now I killed that also).

[root@ser ~]# netstat -tunlpa
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      118/sshd            
tcp        0      0 xx.xx.xx.xx:34784    204.44.101.130:21       ESTABLISHED 1126/sh             
tcp        0     60 xx.xx.xx.xx:22       MYIP:42584     ESTABLISHED 750/sshd: givi [p 
tcp6       0      0 :::22                   :::*                    LISTEN      118/sshd            

I am not sure how to find the culprit script/IP(except the one in netstat). I have tried installing iftop,iptraf-ng,nload but didn't help. Can someone help me with some tool or how to find the source of this issue?

Answer 1

You are describing known signatures of malware in your own answer.

You server was hacked either by a human or a bot. That is malware left behind.

They surely have rootkits where you might even not see what they are running, besides the odd ps of who command. rootiks are able to hide processes and ports open when investigating the compromised system in the command line.

Reinstall from scratch, you do not have any guarantees for what was changed. Change any passwords of users that are on that server, at least their password hashes were compromised, at work a couple of them have already their plain passwords compromised.

Answer 2

I had some help and was able to solve this issue. There was entry in /etc/crontab file like below and the script was the one causing issue.

*/3 * * * * root /etc/cron.hourly/gcc.sh

It was executing some binary(encrypted) which was responsible in creating binaries in /usr/bin with wired names like ipjihmyzuh, fdmhhxthxy, fojoouuupy, fgqzvgtjan, bsitumzdkd, jrddsxyemy, nmwgbaforc, cuveaysdzg.

Let me briefly explain how we nailed this.

From the netstat, I was able to get the PID of that(above case it was 1126/sh, but the name and PID keep on changing).

With lsof I was able to see the binaries with above names.

lsof -p 1126 

We have removed the binaries created and they keep on creating after that with different name. So, as a blind shot, we then checked in the cron related files and just as a try stopped cron and killed every suspicious process and removed binaries again. Binary creation stopped and nothing is happening now.

Started cron again and things got back. Removed that entry from /etc/crontab, removed all newly created binaries, killed all suspicious process and things are normal now.

I am not sure how /etc/crontab got edited or /etc/cron.hourly/gcc.sh got placed there. The guy who helped me said that most probably happened as root user might got hacked somehow. I have disabled ssh root authentication and using only sudo user now.