I have a server, it's outdated running Debian Lenny, and yes I know that is probably half the problem. It also has PHPMyAdmin and ProFTPd. Again, I get it, all bad signs.

But for the life of me, I cannot figure out how this user is logging in and adding files and executing commands.

They are able to start screen sessions, and type things like nano file.sh and then create a script and ./file.sh to execute it. Does this mean they have SSH access? I don't understand.

I check all of my log files, and nothing anywhere shows successful authentication. I check users, who, last, every little command I can type - nothing shows any signs of someone being logged in.

Every now and again, I notice they create new directories and the owner is 500 or 1XXX, but these accounts don't show up when I look for them.

Is there something I can do to figure out wtf is going on? We are going to wipe the server clean, don't get me wrong, but I'd like to know what happened exactly so I can avoid this sort of problem in the future.

I don't want any recommendations regarding "don't use phpmyadmin, old unsupported distros, ftp, etc.", on our new server we won't have anything insecure, and will use passworded SSH Auth keys, etc.

I just want a bit of insight on how I can know when the user is logged in, and where they logged in from. Granted, I'm probably not giving enough information, but maybe something will click for someone? Thanks.

  • 1
    Check /etc/passwd to see if there are any new users. If a user makes a file or directory and then the account is deleted, the owner/group is ls -la usually shows up as a number instead of a name.
    – cutrightjm
    Jan 28 '16 at 17:58
  • Please check my answer here security.stackexchange.com/questions/111325/… Jan 28 '16 at 19:20
  • Do not trust who and last blindly. I can log in with a regular account in any Linux or FreeBSD system with appearing there... Jan 28 '16 at 19:22

Most scripted and manual break-ins do:

  • clean up log entries and similar traces of the break-in
  • install a rootkit, which allows entry to the system outside of default server programs
  • replace default programs (like ps, netstat, ls, etc.) with manipulated versions which hide any activity of the above mentioned rootkit (ie. ps won't show the running rootkit process)

Sometimes those attacks are faulty and do leave traces behind. But in any case: you cannot trust any diagnostic tools you have on the system.

If you want to play a bit around and learn you could:

  1. Install and run 'rkhunter' [*] for example, which checks for known rootkits, but you cannot trust the output without:

    • having run it at least once before the break-in happened
    • hoping that the attacker ignored a rkhunter install on the system (did not manipulate rkhunter itself)
  2. Boot from a rescue CD/USB

    • Mount the systems disks and look around with the binaries of the rescue system
    • comparing md5sums of binaries with the stock version.
    • load the system into a VM and inspect the network traffic

tl;dr: It is near impossible to find out the attack vector on such an open system. One way or another:

Please be responsible and take the system off the internet ASAP and set it up newly from scratch.

[*] or other IDS systems, there are many.

  • 1
    Taking the system offline for good today, and i'm actively monitoring to make sure nothing bad is happening until that time. Waiting for our new server to be provisioned - and we still need our database up.
    – daemon
    Jan 28 '16 at 17:45
  • Please be aware that the current state when connected to the internet is, that someone else is root and you are a user, at the same time you are responsible (ie. imagine it could be a staging system to attack some nuclear research facility)
    – Thomas
    Jan 28 '16 at 17:50
  • understood, but so far it has only been nmap scans of ips in numerical order, and looking for more systems. seems they are trying to accumulate as many open systems as possible and create a botnet. does not seem they are trying to harness my power to start ww3, but i do take this seriously, which is why i haven't slept and it's nearly 1pm - as i actively monitor. (also, i keep deleting nmap and any other tool they install)
    – daemon
    Jan 28 '16 at 17:53
  • 1
    +1 just for "you cannot trust any diagnostic tools you have on the system.". Take the system off line and diagnose it from a different OS (e.g. a liveCD). If you do not have the option to use a spare or to replace disks then boot from a live CD (or I guess a live USB these days), dd & natcat the disk image to someplace to research it later and then nuke from orbit.
    – Hennes
    Jan 28 '16 at 18:33

Even if your new server operating system is hardened to the nth degree, and you use the latest patched software and have no more plain text services; they just may have gained access on the old server through the web application running on the server - for example a SQL injection or file upload vulnerability. So you may get owned again if you deploy the same application on the new server.

So I really would recommend you code review (though I also recommend 'and') perform a web application penetration test of the web application using someone who does that kind of testing professionally/ offer a bug bounty to the community to do it for you. And that you carry on patching, vulnerability scanning and pen testing on a regular basis - at least once a year or on software change depending, of course, on how much you value the service the server/application provides you.

PS whilst your server is off line, consider taking its image and running autopsy against the image to generate a time line for the hack(s) - concentrate of the user ids that you saw 500, 1xxx. I suspect you will find it useful to know how forensics are performed especially if you are likely to be targeted again. Though as pointed out in earlier answers the hack may have covered their tracks and changed key libraries and executables to leave a false time line. Again you can MD5sum the executables and libraries in the image and compare with original versions to find out if they bothered to cover their tracks.


Let's get paranoid ! First of all, shut down this linux and use dd command to make a complete copy of the disks.

The system is directly connected to the internet ? If no, that good, you can start sniffing it's traffic, so we can determine some useful things. If the answer is yes, try to put some machine between it and the internet. Why not sniff directly inside the machine ? With a rootkit would be possible to become invisible to sniffing, and our efforts will be a waste of time.

I strongly recommend that you extract all your information, code, databases, and manually review everything to make sure that there is no backdoors on it.

Reconstruct a similar machine, with the same version of every lib, and then scan the whole system, calculating the hash of everything, use SHA1 or SHA2, with MD5 we can have collisions easily calculated, we are paranoids, remember that ! Compare both lists, plus, try to detect any unusual parameter on grub/lilo, any strange module loading.

Check users and ssh keys too. Check your source code for bugs, for example, he can run code as apache, and leave a backdoor to escalate privileges to root user.

All checks must be done using the images of the disk, in a offline environment.

Getting back to your sniffing session, try to detect any unusual traffic. If your machine was compromised possibly it will be used to something illegal, like DDoSing or being a pivot for another attack.

Once you detect from where the attacker connect from (I bet that he uses TOR), and how he connects, you can start to trace him. Determine if it's an automated script or and real user that is controlling your server. You can download and change bash/sh/zsh or what shell your attacker is using to save logs even if the user unset the variable. Do the same with SSH, alter the code to create somewhere else, a log with processes started, and logins. If he connects using a backdoor (That's what is most probable), try to determine how it works, and determine what he is using your systems for.

Why do it ? Because if he is doing something illegal, sooner os later someone will trace him, and will find you in the way. Best way to prevent yourself to get into some serious trouble, is get evidence of what is happening, and after that, try to shut down this environment, and get a good lawyer.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.