2

I recently started migrating some of my company's servers to Google's compute engine. Among other things I set up a 2-node elasticsearch cluster.

Today I run top command on one of the nodes and I noticed a few suspicious processes:

 PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                     
11995 elastics  20   0  424m  25m  400 S 533.0  0.1 877:26.82 gg                                                                                                         
30494 elastics  20   0 23.4g  16g 116m S  0.8 56.7   2:49.61 java                                                                                                       
20148 newrelic  20   0  244m 5824 2872 S  0.5  0.0   9:34.29 nrsysmond                                                                                                   
42 root      20   0     0    0    0 S  0.3  0.0   0:41.16 events/7                                                                                                   
4 root      20   0     0    0    0 S  0.1  0.0   0:54.41 ksoftirqd/0                                                                                                 
38 root      20   0     0    0    0 S  0.1  0.0   0:22.52 events/3                                                                                                    
39 root      20   0     0    0    0 S  0.1  0.0   0:20.27 events/4
2876 root      20   0 15152 1312  928 R  0.1  0.0   0:00.10 top
7022 elastics  20   0  424m  16m  380 S  0.1  0.1   1:10.45 freeBSD
1 root      20   0 19340 1312  952 S  0.0  0.0   0:02.38 init
2 root      20   0     0    0    0 S  0.0  0.0   0:00.12 kthreadd
3 root      RT   0     0    0    0 S  0.0  0.0   0:01.44 migration/0
5 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 stopper/0        

The processes with ids 11995, 7022 seemed suspicious and I decided to have a better look.

Doing an lsof for this processes uncovered that they had established connections to either China or Hong Kong.

Please find an example snippet below:

[root@elastic-prd-node-01 fail2ban]# lsof -p 11995
COMMAND   PID          USER   FD   TYPE   DEVICE SIZE/OFF  NODE NAME
gg      11995 elasticsearch  cwd    DIR      8,1     4096     2 /
gg      11995 elasticsearch  rtd    DIR      8,1     4096     2 /
gg      11995 elasticsearch  txt    REG      8,1  1415201 16117 /tmp/gg (deleted)
gg      11995 elasticsearch  mem    REG      8,1 99154480  4784 /usr/lib/locale/locale-archive
gg      11995 elasticsearch    0u   CHR      1,3      0t0  3881 /dev/null
gg      11995 elasticsearch    1u   CHR      1,3      0t0  3881 /dev/null
gg      11995 elasticsearch    2u   CHR      1,3      0t0  3881 /dev/null
gg      11995 elasticsearch    3u  IPv4 10615541      0t0   TCP myservername.c.myproject.internal:54431->103.206.22.224:28099 (ESTABLISHED)
gg      11995 elasticsearch    4u  IPv4 10656374      0t0   UDP *:49882 
gg      11995 elasticsearch    5u  IPv4 10656375      0t0   UDP *:41943 
gg      11995 elasticsearch    6u  IPv4 10656376      0t0   UDP *:49792 
gg      11995 elasticsearch    7u  IPv4 10656377      0t0   UDP *:32849 
gg      11995 elasticsearch    8u  IPv4 10656378      0t0   UDP *:35841 
gg      11995 elasticsearch    9u  IPv4 10656379      0t0   UDP *:52037 
gg      11995 elasticsearch   10u  IPv4 10656380      0t0   UDP *:45405 
gg      11995 elasticsearch   11u  IPv4 10656381      0t0   UDP *:55345 
gg      11995 elasticsearch   12u  IPv4 10656382      0t0   UDP *:49990 
gg      11995 elasticsearch   13u  IPv4 10656383      0t0   UDP *:34888 
gg      11995 elasticsearch   14u  IPv4 10656384      0t0   UDP *:46642 
gg      11995 elasticsearch   15u  IPv4 10656385      0t0   UDP *:38881 
gg      11995 elasticsearch   16u  IPv4 10656386      0t0   UDP *:38455 
gg      11995 elasticsearch   17u  IPv4 10656387      0t0   UDP *:47360 
gg      11995 elasticsearch   18u  IPv4 10656388      0t0   UDP *:40310 
gg      11995 elasticsearch   19u  IPv4 10656389      0t0   UDP *:44127 
gg      11995 elasticsearch   20u  IPv4 10656390      0t0   UDP *:46064 
gg      11995 elasticsearch   21u  IPv4 10656391      0t0   UDP *:55830 
gg      11995 elasticsearch   22u  IPv4 10656392      0t0   UDP *:43110 
gg      11995 elasticsearch   23u  IPv4 10656393      0t0   UDP *:58793 

I also ran:

[root@elastic-prd-node-01 tmp]# ls -al /proc/11995/exe
lrwxrwxrwx. 1 elasticsearch elasticsearch 0 May  8 12:48 /proc/17525/exe -> /tmp/gg

and I noticed that in the /tmp directory there were a few suspicious files named like: gg or freeBSD or syn2016.

I then tarred the whole /tmp directory, killed all the suspicious processes, removed all the suspicious files from /tmp and shutdown the servers.

What I should do next? Does this look normal? Does it look like an attack? What I should do in order to prevent it happening in the future? What steps can I take in order to get to the bottom of it?

p.s. I am running elasticsearch 1.6.2 on CentOS 6 and I used elasticdump to migrate my indices.

1

This is definitely an attack. Someone managed to

a) drop a file onto your server (possibly via poorly configured apache)
b) execute that file

You should do:

a) mount the /tmp partition with the noexec - option
b) reboot the server
c) run a malware scan on the complete host while offline
d) if that does not help : REINSTALL from scratch

To get to the bottom of it, you can

a) check the gg file's creation date
b) look into the httpd-logs at that time.

So you possibly will be able to see via which script the file has been dropped. Another possibility would be a privileged ftp account with poor password. Search these logs also...

  • I don't have httpd installed on these boxes – Christos May 11 '16 at 15:46
1

Elasticsearch is not secure when open to the internet btw. Not sure if yours is, just thought I should point that out.

"Using a firewall is always important. The elasticsearch http module was not designed to be exposed to untrusted networks as it doesn't provide authorization neither authentication mechanisms. By default, elasticsearch listens on two ports: 9200/tcp (restful api) and 9300/tcp (for java node/transport clients and communications between nodes in a cluster)." (https://groups.google.com/forum/#!topic/ica-atom-users/zkZqqTULvn4)

  • that link was very useful – Christos May 11 '16 at 20:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.