Someone asked if this constituted a break-in and wondered if their server had been compromised.

Jan 12 04:16:51 foo sshd[26725]: Failed password for root from 61.174.51.207 port 1076 ssh2
Jan 12 04:16:54 foo sshd[26822]: Disconnecting: Too many authentication failures for root
Jan 12 04:16:54 foo sshd[26825]: Failed password for root from 61.174.51.207 port 1076 ssh2
Jan 12 04:16:54 foo sshd[27324]: PAM 5 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=61.174.51.207 user=root "secure" 229L, 24376C

How does one go about diagnosing whether their server was compromised?

  • 2
  • 1
    I'm slightly confused, shouldn't this be an edit to the existing question? – Chris Down Jan 13 '14 at 4:16
  • 1
    @warl0ck - I'd say it's not a duplicate since the other is focusing on the mysql bits, I wanted to break this off and make it more generic and provide a checklist of things to do. – slm Jan 13 '14 at 4:17
  • @ChrisDown - I didn't want to rip out his mysql pieces since I think his issue is really just that his provider did an upgrade on him and his services didn't restart. But I wanted to get a Q&A like this into our site. – slm Jan 13 '14 at 4:18
  • @warl0ck - BTW guys, in order to be duplicate the linked Q needs to have an A, that one has none and I don't think it was worth producing a A for it, given it's a poorly asked Q. – slm Jan 13 '14 at 4:29
up vote 6 down vote accepted

False alarms

The above log shows that someone was attempting to break into the system but they were unsuccessful. This line shows that they tried and failed 5 times to SSH into the system as root.

PAM 5 more authentication failures...

If you see these kinds of messages in your logs it's good to investigate them so that you understand what they mean, but also not to get too alarmed by them either.

NOTE: This type of chatter in logs is often referred to as IBR (Internet Background Radiation) or IBN (Internet Background Noise).

Convinced you've been hacked? -- How to diagnose

Having been someone who has worked with the FBI one time before, in my past, these were the following things I did to diagnose a server that had been compromised. These are in no particular order!

  1. Take the system down immediately.

  2. If you suspect a system has been compromised, you can no longer trust any of the software on this server. So use alternative tools, i.e. either shutdown the server and mount the disk slaved, or boot from a known good live CD/USB.

  3. Logs may have been tampered with, but begin your analysis by scrutinizing them for anomalies.

  4. Archive the system. If you haven't been backing it up, do so now.

  5. Reverse engineer how the attackers got in so that you understand the vulnerabilities and can block them in the future.

  6. Keep an archived set of the system, on the off chance that law enforcement officials may show up asking you information about your system.

  7. Reverse engineer any back doors and/or software the attackers installed so that you can understand what nefarious purposes your system was used for by the attackers.

  8. Perform your analysis using the copied data.

  9. When you reinstall, make sure that you aren't reinstalling compromised data or vulnerable code.

  • I would also point out that there is no standard way of determining whether a server is compromised. It's just the nature of system security. If there were a way to detect it that worked for every scenario, security analysts would be out of a job :-). I would also argue against point #1 in this case. The ssh log entries provided are very common for a publicly accessible ssh server. If you took the system down every time you saw them, you'd have more downtime than uptime. You just have to use your judgement. – Patrick Jan 13 '14 at 4:16
  • @Patrick - thanks for the feedback. I was trying to split this as 2 parts. The 1st discusses that the logs are normal. The 2nd was steps you can do if you think there was a breakin. – slm Jan 13 '14 at 4:20
  • @Patrick - does that help? I tried to split the A a bit more via the headings. – slm Jan 13 '14 at 4:23
  • Wouldn't 1 be a memory dump? dd if=/dev/fmem of=mem … – Runium Jan 13 '14 at 4:27
  • @Sukminder - sure, note those are in no particular order. I said that right before giving the list. – slm Jan 13 '14 at 4:32

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.