I'm going to write an answer with RHEL in mind but just know that if you're on a SuSE or Debian-based distro there are going to be analogs for what I describe.
First off, if you just want to verify that it was a system update and not someone trying to rootkit the machine, you can "verify" the openssh-clients
package like so:
[root@hypervisor scsd]# rpm -V openssh-clients
[root@hypervisor scsd]#
[root@hypervisor scsd]# rpm -V openssh-server
S.5....T. c /etc/pam.d/sshd
S.5....T. c /etc/ssh/sshd_config
[root@hypervisor scsd]#
I did openssh-server
as well so you can see what it looks like when something changed. The important part is the "5" which tells us that the md5sum of the file is different than what exists in the RPM database. If that checks out it was probably due to a system update.
If they used yum (highly likely) there will be a /var/log/yum.log entry for that RPM being updated. This is useful for getting the specific time the update occurred for later review with yum history
.
If they used rpm
directly you can either do some queryformat
magic or rpm -q openssh-clients --last
to get the date it happened (although it sounds like you already know that bit of information)
There is a yum
subcommand called history
that records the invoking user's auid/loginuid:
[root@hypervisor scsd]# yum history
Loaded plugins: product-id, refresh-packagekit, rhnplugin, security, subscription-manager
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
This system is receiving updates from RHN Classic or RHN Satellite.
ID | Login user | Date and time | Action(s) | Altered
-------------------------------------------------------------------------------
54 | <jadavis6> | 2013-07-15 09:03 | Install | 2
53 | <jadavis6> | 2013-07-09 17:25 | Update | 23
52 | <jadavis6> | 2013-06-24 10:10 | Install | 3 <
51 | <jadavis6> | 2013-06-14 22:33 | Install | 1 >
50 | <jadavis6> | 2013-06-14 07:47 | E, I, U | 90
49 | root <root> | 2013-06-14 00:58 | Update | 1
48 | <jadavis6> | 2013-06-03 08:28 | Install | 3
47 | <jadavis6> | 2013-05-28 11:57 | Install | 3 <
46 | <jadavis6> | 2013-05-20 18:25 | Install | 1 >
45 | <jadavis6> | 2013-05-20 12:00 | Install | 1
44 | <jadavis6> | 2013-05-19 15:29 | Install | 6
43 | <jadavis6> | 2013-05-18 20:16 | Install | 3
42 | <jadavis6> | 2013-05-16 16:21 | Install | 2 <
41 | <jadavis6> | 2013-05-16 12:48 | Install | 1 >
40 | <jadavis6> | 2013-05-10 09:28 | Install | 1
39 | <jadavis6> | 2013-05-10 09:28 | Install | 1
38 | <jadavis6> | 2013-04-29 19:45 | Install | 2 <
37 | <jadavis6> | 2013-04-29 18:51 | Install | 8 >
36 | <jadavis6> | 2013-04-29 18:35 | Update | 11
35 | <jadavis6> | 2013-04-27 15:44 | E, I, O, U | 429 EE
history list
[root@hypervisor scsd]#
The loginuid is unforgable because as children of init are spun off they start with a loginuid of -1 (negative one). When you log in via a tty or sshd
pam_loginuid (there are other modules that also do this) sets it to the authenticated user's UID. Once set to something other than -1
even root can not change this value (only in newer kernels, though) since it's non-functional/accounting-only and needs to take into consideration that an attacker may have gained root. All children inherit the parent's loginuid so even though sudo
spawns a program with the EUID of zero (or whichever user) you're still going to have the same loginuid.
You can test this by just doing your share of sudo's and doing a cat /proc/self/loginuid
each time it should the user you initially logged in as no matter how many su
or sudo
invocations you've done since. This is how yum history
up there knows jadavis6
did the yum update
even though I did them all as the root user.
If there's some ambiguity between two yum transactions you can do a yum history info <transID>
like if I wanted to know more about that last transaction:
[root@hypervisor scsd]# yum history info 35
Loaded plugins: product-id, refresh-packagekit, rhnplugin, security,
: subscription-manager
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
This system is receiving updates from RHN Classic or RHN Satellite.
Transaction ID : 35
Begin time : Sat Apr 27 15:44:57 2013
Begin rpmdb : 959:3d300ae2e8dc239f9f972306ba2406bd22ba29bc
End time : 15:50:39 2013 (5 minutes)
End rpmdb : 972:381cb76592ea2f779ee4521a4e7221196520486a
User : <jadavis6>
Return-Code : Success
Command Line : update -y
Transaction performed with:
Updated rpm-4.8.0-27.el6.x86_64 @anaconda-RedHatEnterpriseLinux-201206132210.x86_64/6.3
Updated subscription-manager-0.99.19.4-1.el6_3.x86_64 @rhel-x86_64-server-6
Updated yum-3.2.29-30.el6.noarch @anaconda-RedHatEnterpriseLinux-201206132210.x86_64/6.3
Installed yum-metadata-parser-1.1.2-16.el6.x86_64 @anaconda-RedHatEnterpriseLinux-201206132210.x86_64/6.3
Packages Altered:
Updated NetworkManager-glib-1:0.8.1-34.el6_3.x86_64 @rhel-x86_64-server-6
Update 1:0.8.1-43.el6.x86_64 @rhel-x86_64-server-6
Updated PackageKit-0.5.8-20.el6.x86_64 @rhel-x86_64-server-6
Update 0.5.8-21.el6.x86_64 @rhel-x86_64-server-6
Updated PackageKit-device-rebind-0.5.8-20.el6.x86_64 @rhel-x86_64-server-6
[...snip...]
Updated yum-3.2.29-30.el6.noarch @anaconda-RedHatEnterpriseLinux-201206132210.x86_64/6.3
Update 3.2.29-40.el6.noarch @rhel-x86_64-server-6
Updated yum-rhn-plugin-0.9.1-40.el6.noarch @anaconda-RedHatEnterpriseLinux-201206132210.x86_64/6.3
Update 0.9.1-43.el6.noarch @rhel-x86_64-server-6
Scriptlet output:
1 warning: /etc/shadow created as /etc/shadow.rpmnew
2 No input from event server.
3 warning: %postun(wdaemon-0.17-2.el6.x86_64) scriptlet failed, exit status 1
history info
I truncated it because it looks like this one was a pretty lengthy system update.
AFAIK there's no way to trace it back if they updated via rpm
directly outside of configuring auditd
to monitor changes to the RPM database files. Doing so would let you do an ausearch
will let you see a list of PID's that made changes and the loginuid associated (which will be displayed as auid
).
If they directly changed it outside of RPM, you would have to be watching each of the files you wanted to monitor for changes before the change was made. After the fact you can't do much but you might consider doing something with auditd
to monitor files you feel are rootkit targets. Doing too much can bog the system down. It's also worth mentioning that you should ship these logs off-server somewhere to prevent malicious tampering.
Hope that helps.
EDIT:
One thing to note, it looks like root can change the loginuid if it has CAP_SYS_AUDITCONTROL
(not required if the loginuid is currently -1
) but you should be able to remove that capability from the system's bounding set, which would require the attacker to reboot the system to gain that capability, which is a noisy as hell operation that leaves auditable events all over the place so they're unlikely to do that.
~/.bash_history
files of these users? If they have any braincells they removed that trail, but you can always check.