6

I spent some time looking at htop and found a row of weird processes/commands executed by the root user for short amounts of time and I was wondering whether this is normal behavior (or some kind of malware). Here is a non exhaustive list of commands that appear in htop. Only Tomcat and MySQL are running on this Debian Server.

ifconfig eth 0
ps -ef
bash
uptime
top
netstat -antop
pwd
echo "find"
gnome-terminal
whoami
sleep 1
id
su
cd /etc

enter image description here

Now this looks evil, doesn't it?
Killing a process like that makes a new one spawn immediately, always involing a random string. enter image description here

  • Just to be clear, neither you nor any other user are executing these commands? How many users does this machine have? – Faheem Mitha Dec 14 '14 at 13:52
  • Just me, there is no other user. And it's always the same commands. Nonsense commands like "grep "A"" (Always with A never with anything else) and "sleep 1" for example. – user94900 Dec 14 '14 at 14:05
  • The grep sounds like it's part of a pipe, from a shell script. I see that crond is running, did you check your cron jobs? – jsbillings Dec 14 '14 at 14:20
  • I used crontab -u XXX on all users and for everyone it says "no crontab for XXX". The /etc/cron.d folder is empty, cron.daily has a few files with seemingly innocent content... – user94900 Dec 14 '14 at 14:28
  • Killing the Cron process didn't help either. – user94900 Dec 14 '14 at 14:32
5

This sure looks like malware that isn't hiding itself very well. Well-written malware would infect the kernel and arrange to hide itself completely from the task list. This one clumsily disguises itself as the innocuous uptime, but does a bad job of it, and uptime is suspicious anyway because it wouldn't be running for such a long time.

If you confirm that this is malware, read How do I deal with a compromised server?

You're going to have to reinstall the system. Even though the malware looks clumsy, you can't be sure that you'll be able to eradicate it: there may be a part of it that's better hidden.

Before you reinstall, try to figure out how the malware got there. Did you install a program from a bad source? Did you install a program with a known security hole? Check the logs, the file dates, the command history, etc.

When you reinstall, make sure to get all software from a clean source. Verify the checksum of your installation media against a checksum on the provider's HTTPS website. Stick to software from the distribution as much as possible and make sure that checksum verification is performed (it is by default under Debian). Privilege binaries from the distribution to binaries of less trustworthy origin. Make sure to apply all security updates before activating any Internet-facing services. If you need to install out-of-distribution software, make sure to retrieve it from a reliable source, to get the latest version with no known security holes, and to give it as few privileges as possible. Use strong passwords (if necessary, write them down on a sticky note next to your monitor if you work in a secure area).

  • I already reset my server, this time using a keyfile instead of a pwd. Maybe that will help. As for how the malware got there makes me wonder, the server was set up two days ago. The root password was the amazing sequence "abcdefgh12345678910" with no additional security activated. I visited the server from home and work with putty and winscp. Apache2, Tomcat and Mysql were running, as for sources, I only added ppa.launchpad.net (Java 8 via apt) and wheezy-backports (tc8). TC8 management was protected with lockout realm. – Traubenfuchs Dec 14 '14 at 23:31
  • @Traubenfuchs Most SSH attacks only try extremely weak passwords because they look for the weakest of the herd, but this password is weak against insistent online attacks and laughable against local attacks. Use strong passwords next time. – Gilles Dec 14 '14 at 23:45
  • I know my password is following a shockingly simple logic, but I always thought brute force programs would stop at 9 at worst and not go for combined numbers like 10, I felt like I was doing a unique thing that brute force programs don't probe for. Anyways, I am using a keyfile like on my other servers now. In the end it was nothing more than laziness. – Traubenfuchs Dec 15 '14 at 10:20
2

You can also use pstree or ps auxf to find out which process is running which. (Maybe that output is more readable.)

As you suspect any malware you should also check for processes trying to communicate on the network.

You can also use netstat -tupln to check if any unexpected process is listening for remote connections. Similarly netstat -tupn will show you the current communication.

Also have a look at the startup scripts in /etc/init.d/ or even /etc/rc.*/ for any unusual entries. Same for crontab as already suggested in comments.

  • I already reset my server and there was indeed a connection opened to a Chinese IP... – Traubenfuchs Dec 14 '14 at 23:27
1

/boot/nnfwcjkwna does not look ok to me.

Check out the process's pid and type:

ls -l /proc/pid-number/exe

Example:

enter image description here

This way you will see the full path of the executable. Go there and check out the contents with ls -al. For binary files viewing use: strings file | more And then use space to browse down.

Check out to see new opened ports and identify their exes with nestat and/or lsof.

As for some security advices:

  1. Go in single user mode runlevel.
  2. Check /etc/passwd and /etc/shadow to be sure that no new users have been added and also cron config file.
  3. Change root password.
  4. Disallow direct root login and "su" command.
  5. Add a new user for you and set it up for sudo as the only user to be able to run sudo.
  6. chmod 000 /usr/bin/sudo and then use a setfacl r-x rule to allow only that user to use sudo.
  7. chattr +iau /etc/passwd /etc/shadow
  8. Check out your init files, it might have been installed there also so it will run on reboot again.
  9. Go back to the usual runlevel and start network.
  10. nmap -v -sS -O 127.0.0.1 -p "1-65500"

Hope at least some of these will help you.

EDIT: As stated bellow this is not a permanent solution, but something to help investigate the problem. It isn't of much use if you directly reinstall the system without actually knowing how everything happened.

  • None of your “security advices” helps against a moderately competent malware. If the malware runs as root, it can infect the kernel, and then passwords and permissions don't matter. – Gilles Dec 14 '14 at 22:46
  • Wanted to edit the post but I will reply here. This isn't a permanent solution of course, you do this if you want to investigate the problem on the server being alive. To see exactly where the malware is located and how did it got there. – jojo621 Dec 14 '14 at 22:52
  • I reset the server already. It clearly was malware. – Traubenfuchs Dec 14 '14 at 23:32
0

According to this solution, your virus is in /lib/libudev.so or /lib/libudev4.so. You need to:

chattr -i /lib/libudev.so 

Then remove, reboot and then delete all other stuff, as described in the link.

You can install ClamAV.

  • both links do not work for me – DarkHeart Feb 15 '17 at 2:54
  • @DarkHeart Now it should be ok. My bad with the first one. The second one's been ok I gather. – Tomasz Feb 15 '17 at 3:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.