False alarms
The above log shows that someone was attempting to break into the system but they were unsuccessful. This line shows that they tried and failed 5 times to SSH into the system as root.
PAM 5 more authentication failures...
If you see these kinds of messages in your logs it's good to investigate them so that you understand what they mean, but also not to get too alarmed by them either.
NOTE: This type of chatter in logs is often referred to as IBR (Internet Background Radiation) or IBN (Internet Background Noise).
Convinced you've been hacked? -- How to diagnose
Having been someone who has worked with the FBI one time before, in my past, these were the following things I did to diagnose a server that had been compromised. These are in no particular order!
Take the system down immediately.
If you suspect a system has been compromised, you can no longer trust any of the software on this server. So use alternative tools, i.e. either shutdown the server and mount the disk slaved, or boot from a known good live CD/USB.
Logs may have been tampered with, but begin your analysis by scrutinizing them for anomalies.
Archive the system. If you haven't been backing it up, do so now.
Reverse engineer how the attackers got in so that you understand the vulnerabilities and can block them in the future.
Keep an archived set of the system, on the off chance that law enforcement officials may show up asking you information about your system.
Reverse engineer any back doors and/or software the attackers installed so that you can understand what nefarious purposes your system was used for by the attackers.
Perform your analysis using the copied data.
When you reinstall, make sure that you aren't reinstalling compromised data or vulnerable code.