I'm the main sysadmin of a system. In the system there are 3 sudoers users with root's privileges.
The system runs a script in background which check the hash of system's utilities to detect possible malicious changes. Today I was alerted that the ssh utility's hash changed.
There weren't any updates till now, and I think one of sudoers is responsible for this. Is it possible to detect which sudoer changed the ssh utility?
I'm going to write an answer with RHEL in mind but just know that if you're on a SuSE or Debian-based distro there are going to be analogs for what I describe.
First off, if you just want to verify that it was a system update and not someone trying to rootkit the machine, you can "verify" the openssh-clients package like so:
[root@hypervisor scsd]# rpm -V openssh-clients
[root@hypervisor scsd]#
[root@hypervisor scsd]# rpm -V openssh-server
S.5....T. c /etc/pam.d/sshd
S.5....T. c /etc/ssh/sshd_config
[root@hypervisor scsd]#
I did openssh-server as well so you can see what it looks like when something changed. The important part is the "5" which tells us that the md5sum of the file is different than what exists in the RPM database. If that checks out it was probably due to a system update.
If they used yum (highly likely) there will be a /var/log/yum.log entry for that RPM being updated. This is useful for getting the specific time the update occurred for later review with yum history.
If they used rpm directly you can either do some queryformat magic or rpm -q openssh-clients --last to get the date it happened (although it sounds like you already know that bit of information)
There is a yum subcommand called history that records the invoking user's auid/loginuid:
[root@hypervisor scsd]# yum history
Loaded plugins: product-id, refresh-packagekit, rhnplugin, security, subscription-manager
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
This system is receiving updates from RHN Classic or RHN Satellite.
ID | Login user | Date and time | Action(s) | Altered
-------------------------------------------------------------------------------
54 | <jadavis6> | 2013-07-15 09:03 | Install | 2
53 | <jadavis6> | 2013-07-09 17:25 | Update | 23
52 | <jadavis6> | 2013-06-24 10:10 | Install | 3 <
51 | <jadavis6> | 2013-06-14 22:33 | Install | 1 >
50 | <jadavis6> | 2013-06-14 07:47 | E, I, U | 90
49 | root <root> | 2013-06-14 00:58 | Update | 1
48 | <jadavis6> | 2013-06-03 08:28 | Install | 3
47 | <jadavis6> | 2013-05-28 11:57 | Install | 3 <
46 | <jadavis6> | 2013-05-20 18:25 | Install | 1 >
45 | <jadavis6> | 2013-05-20 12:00 | Install | 1
44 | <jadavis6> | 2013-05-19 15:29 | Install | 6
43 | <jadavis6> | 2013-05-18 20:16 | Install | 3
42 | <jadavis6> | 2013-05-16 16:21 | Install | 2 <
41 | <jadavis6> | 2013-05-16 12:48 | Install | 1 >
40 | <jadavis6> | 2013-05-10 09:28 | Install | 1
39 | <jadavis6> | 2013-05-10 09:28 | Install | 1
38 | <jadavis6> | 2013-04-29 19:45 | Install | 2 <
37 | <jadavis6> | 2013-04-29 18:51 | Install | 8 >
36 | <jadavis6> | 2013-04-29 18:35 | Update | 11
35 | <jadavis6> | 2013-04-27 15:44 | E, I, O, U | 429 EE
history list
[root@hypervisor scsd]#
The loginuid is unforgable because as children of init are spun off they start with a loginuid of -1 (negative one). When you log in via a tty or sshd pam_loginuid (there are other modules that also do this) sets it to the authenticated user's UID. Once set to something other than -1 even root can not change this value (only in newer kernels, though) since it's non-functional/accounting-only and needs to take into consideration that an attacker may have gained root. All children inherit the parent's loginuid so even though sudo spawns a program with the EUID of zero (or whichever user) you're still going to have the same loginuid.
You can test this by just doing your share of sudo's and doing a cat /proc/self/loginuid each time it should the user you initially logged in as no matter how many su or sudo invocations you've done since. This is how yum history up there knows jadavis6 did the yum update even though I did them all as the root user.
If there's some ambiguity between two yum transactions you can do a yum history info <transID> like if I wanted to know more about that last transaction:
[root@hypervisor scsd]# yum history info 35
Loaded plugins: product-id, refresh-packagekit, rhnplugin, security,
: subscription-manager
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
This system is receiving updates from RHN Classic or RHN Satellite.
Transaction ID : 35
Begin time : Sat Apr 27 15:44:57 2013
Begin rpmdb : 959:3d300ae2e8dc239f9f972306ba2406bd22ba29bc
End time : 15:50:39 2013 (5 minutes)
End rpmdb : 972:381cb76592ea2f779ee4521a4e7221196520486a
User : <jadavis6>
Return-Code : Success
Command Line : update -y
Transaction performed with:
Updated rpm-4.8.0-27.el6.x86_64 @anaconda-RedHatEnterpriseLinux-201206132210.x86_64/6.3
Updated subscription-manager-0.99.19.4-1.el6_3.x86_64 @rhel-x86_64-server-6
Updated yum-3.2.29-30.el6.noarch @anaconda-RedHatEnterpriseLinux-201206132210.x86_64/6.3
Installed yum-metadata-parser-1.1.2-16.el6.x86_64 @anaconda-RedHatEnterpriseLinux-201206132210.x86_64/6.3
Packages Altered:
Updated NetworkManager-glib-1:0.8.1-34.el6_3.x86_64 @rhel-x86_64-server-6
Update 1:0.8.1-43.el6.x86_64 @rhel-x86_64-server-6
Updated PackageKit-0.5.8-20.el6.x86_64 @rhel-x86_64-server-6
Update 0.5.8-21.el6.x86_64 @rhel-x86_64-server-6
Updated PackageKit-device-rebind-0.5.8-20.el6.x86_64 @rhel-x86_64-server-6
[...snip...]
Updated yum-3.2.29-30.el6.noarch @anaconda-RedHatEnterpriseLinux-201206132210.x86_64/6.3
Update 3.2.29-40.el6.noarch @rhel-x86_64-server-6
Updated yum-rhn-plugin-0.9.1-40.el6.noarch @anaconda-RedHatEnterpriseLinux-201206132210.x86_64/6.3
Update 0.9.1-43.el6.noarch @rhel-x86_64-server-6
Scriptlet output:
1 warning: /etc/shadow created as /etc/shadow.rpmnew
2 No input from event server.
3 warning: %postun(wdaemon-0.17-2.el6.x86_64) scriptlet failed, exit status 1
history info
I truncated it because it looks like this one was a pretty lengthy system update.
AFAIK there's no way to trace it back if they updated via rpm directly outside of configuring auditd to monitor changes to the RPM database files. Doing so would let you do an ausearch will let you see a list of PID's that made changes and the loginuid associated (which will be displayed as auid).
If they directly changed it outside of RPM, you would have to be watching each of the files you wanted to monitor for changes before the change was made. After the fact you can't do much but you might consider doing something with auditd to monitor files you feel are rootkit targets. Doing too much can bog the system down. It's also worth mentioning that you should ship these logs off-server somewhere to prevent malicious tampering.
Hope that helps.
EDIT:
One thing to note, it looks like root can change the loginuid if it has CAP_SYS_AUDITCONTROL (not required if the loginuid is currently -1) but you should be able to remove that capability from the system's bounding set, which would require the attacker to reboot the system to gain that capability, which is a noisy as hell operation that leaves auditable events all over the place so they're unlikely to do that.
If you take a look through the /var/log/audit/audit.log file you should be able to line up the time/date that the ssh file was changed with when one of the 3 of the users with sudoers privileges logged in.
The audit.log file contains lines like this:
type=USER_START msg=audit(1374006520.730:480): user pid=28303 uid=0 auid=500 subj=user_u:system_r:unconfined_t:s0 msg='PAM: session open acct="root" : exe="/usr/bin/sudo" (hostname=?, addr=?, terminal=/dev/pts/7 res=success)'
type=CRED_ACQ msg=audit(1374006535.446:488): user pid=28352 uid=0 auid=500 subj=user_u:system_r:unconfined_t:s0 msg='PAM: setcred acct="root" : exe="/usr/bin/sudo" (hostname=?, addr=?, terminal=/dev/pts/7 res=success)'
type=USER_START msg=audit(1374006535.448:489): user pid=28352 uid=0 auid=500 subj=user_u:system_r:unconfined_t:s0 msg='PAM: session open acct="root" : exe="/usr/bin/sudo" (hostname=?, addr=?, terminal=/dev/pts/7 res=success)'
You can backtrack the AUID (Affective User ID - aka. the user that ran the sudo command).
So AUID 500 is this guy on my system:
$ grep 500 /etc/passwd
sam:x:500:500:Sam M.:/home/sam:/bin/bash
Now the audit.log can be grepped but it's much easier to use the tool ausearch to scan through it. For one thing it will print the audit log's timestamp in a more human readable form:
$ ausearch -x /usr/bin/sudo
...
time->Thu Jul 18 14:41:48 2013
type=CRED_ACQ msg=audit(1374172908.936:45): user pid=21252 uid=0 auid=500 subj=user_u:system_r:unconfined_t:s0 msg='PAM: setcred acct="root" : exe="/usr/bin/sudo" (hostname=?, addr=?, terminal=/dev/pts/5 res=success)'
----
time->Thu Jul 18 14:41:48 2013
type=USER_START msg=audit(1374172908.937:46): user pid=21252 uid=0 auid=500 subj=user_u:system_r:unconfined_t:s0 msg='PAM: session open acct="root" : exe="/usr/bin/sudo" (hostname=?, addr=?, terminal=/dev/pts/5 res=success)'
Here I'm searching through the log file looking for matches that contain the executable sudo, (-x /usr/bin/sudo).
Have you checked syslog? According to man sudo: sudo can log both successful and unsuccessful attempts (as well as errors) to syslog(3), a log file, or both. By default
sudo will log via syslog(3) but this is changeable at configure time or via the sudoers file.
I would think that, unless they have intentionally deleted the log file you should be able to get the info there. To narrow down the time frame you can check the modified date stamp on the SSH util.
~/.bash_historyfiles of these users? If they have any braincells they removed that trail, but you can always check.