Stop excessive logging of SUDO & SSH logins to syslog & wtmp for a specific user account

Question

I have a monitoring server (‘ClusterControl’) which uses a regular user account to login using an ‘interactive shell’ and runs some sudo maintenance commands.

This account logs-in via an SSH Key (passwordless) 4 times per minute it seems.

The /var/log/wtmp, /var/log/auth.log and /var/log/syslog fill up with thousands of login and sudo entries.

Three problems arise:

1. Syslog auditing fails since /var/log/syslog is un-usable without passing it through various grep filters to mask out login entries.

2. System disk runs out of space. The files /var/log/wtmp, /var/log/auth.log and /var/log/syslog get very large in size very quickly and cause serious storage space issues.

3. The last account auditing command effectively, experiences a denial of service since it is rendered un-usable as the serviceaccount username is logged and displayed thousands of times and other account logins are impossible to see.

Question

How can I exclude this user account from being logged?

P.S. I have tried adding a PAM entry of systemd [success=ok default=1] pam_succeed_if.so user in username in the various pam files located in /etc/pam.d folder as some articles have directed but it has not worked.

I’m not sure which file to use and what the correct syntax should be.

/var/log/syslog shows the ssh username login logged by ‘systemd’ at the beginning of the log line.

The server is Ubuntu 18.04

EDIT

ClusterControl runs an interactive shell using ssh -t when it logs in as stated by one of the company’s System Support Engineers in a blog post here. However, their suggestion is merly to enable more frequent log rotations.

Answer 1

There are 2 choices.

TL;DR

1. Syslog Filters

Using rsyslog filters is by far the quickest solution but not as comprehensive as using PAM.

At the top of your rsyslog default conf file, add the keywords you wish to filter on and the filter command. In this case, the filter command is delete ('~').

$ vi /etc/rsyslog.d/50-default.conf
:msg, contains, "clustercontrol" ~
:msg, contains, "pam_unix(sudo:session): session closed for user root" ~
:msg, contains, "Removed session" ~

2. Manage Log Files

While I managed to stop thousands of logs being recorded per day in syslog & auth.log there didn't seem to be a way to stop logging the service account SSH logins from clustercontrol to the /var/log/wtmp and /var/log/lastlog files.

For this, we use logrotate (See below).

Logrotate

Lastlog + wtmp

Even setting noupdate or nowtmp directives for pam_lastlog.so globally, had no effect on my system so I simply made sure to compress and rotate these logs regularly using the logrotate config below.

The below config will rotate the logs monthly or every 200Mb, whichever comes first. Logs will be compressed and retained for a maximum of 12 months:

$ vi /etc/logrotate.conf
# no packages own wtmp, or btmp -- we'll rotate them here
/var/log/wtmp {
    missingok
    monthly
    create 0664 root utmp
    rotate 12
    size 200M
    compress
}

/var/log/btmp {
    missingok
    monthly
    create 0660 root utmp
    rotate 12
    size 200M
    compress
}

Using PAM

PAM Debug

The first step to understanding what was going on was to enable debug messages. The problem was that on my Ubuntu 18.04, the pam debug messages were not being logged - until I explicitly told rsyslog which file to record debug messages in using:

$ vi /etc/rsyslog.d/50-default.conf
*.debug           /var/log/debug.log

$ service rsyslog restart

pam_succeed_if.so

This is the pam module we will use to write conditional tests using pam. The documentation is a little in-complete, dry and not very verbose on just how the rules work and what all the directives do. After reading a dozen or so articles, here's what I managed to piece together:

Understand who's doing the logging

Inside /var/log/auth.log you'll see the log messages we wish to stop for this user account:

Mar 17 13:05:27 dev1 sshd[18833]: pam_unix(sshd:session): session opened for user clustercontrol by (uid=0)

Mar 17 13:04:25 dev1 sudo: pam_unix(sudo:session): session opened for user root by clustercontrol(uid=0)

From the above logs we can tell that pam_unix.so is the pam authentication module doing the logging. It's being called by two separate services sshd and sudo.

The pam type token tells pam what type of authentication is to be used for this module. In this case, session which are things that should be done before and/or after the user is authenticated.

Responsible PAM Config

So lets find the pam module pam_unix.so responsible for these logs in our system's configuration.

$ cd /etc/pam.d

$ grep 'pam_unix.so' * -in
common-account:17:account       [success=1 new_authtok_reqd=done default=ignore]        pam_unix.so 
common-auth:17:auth     [success=1 default=ignore]      pam_unix.so nullok_secure
common-password:25:password     [success=1 default=ignore]      pam_unix.so obscure sha512
common-session:29:session       required        pam_unix.so 
common-session-noninteractive:30:session        required        pam_unix.so 
runuser:5:session               required        pam_unix.so

From above, we can see that pam_unix.so module for ':session' is in the file common-session on line #29.

IMPORTANT

1. Find the line in /etc/pam.d/sshd that includes the common-session file and add your custom pam_succeed_if.so statement just before it.

2. Now do almost exactly the same for the /etc/pam.d/sudo file, but here our custom rule syntax is a little different.

Custom PAM Rules

Sequence is Important

Just before our pam_unix.so rules are run from the @include common-session file, we place our pam_succeed.so module rule.

sshd rule

$ grep -in include /etc/pam.d/sshd
5:@include common-auth
15:@include common-account
29:@include common-session
32:# This includes a dynamically generated part from /run/motd.dynamic
56:@include common-password

Line 29 in our case.

$ vi /etc/pam.d/sshd +29
session [success=done default=ignore] pam_succeed_if.so quiet service in sshd user = clustercontrol
# Standard Un*x session setup and teardown.
@include common-session

Its easier if we read the rules from back to front:

We check if a user named clustercontrol (user = clustercontrol) is authenticated by sshd (service in sshd) then this rule matches and we run the directive (quiet) which makes logging quiet.

The part [success=done means, if you get a match to this rule, we're done so skip processing the next rules, or skip over the next rule in the pam stack.

This effectively stops sshd logging for this user.

sudo rule

For this I added line #6 to /etc/pam.d/sudo :

$ vi /etc/pam.d/sudo
:set number 

  1 #%PAM-1.0
  2 
  3 session    required   pam_env.so readenv=1 user_readenv=0
  4 session    required   pam_env.so readenv=1 envfile=/etc/default/locale user_readenv=0
  5 
  6 session [success=done default=ignore] pam_succeed_if.so quiet uid = 0 ruser = clustercontrol
  7 @include common-auth
  8 @include common-account
  9 @include common-session-noninteractive

session [success=done default=ignore] pam_succeed_if.so quiet uid = 0 ruser = clustercontrol

Our syntax is a little different here. Sudo generally runs commands as the root user (unless special directives tell it otherwise). On unix like systems that's user 0.

So our custom rule is saying, if the sudo command is run in the context of user with uid 0 by the remote user (ruser), in our case this is clustercontrol, then be quiet with logging and we're done so skip any further rules for this user of type session authentication.

NB
Don't forget to comment out the *.debug line in /etc/rsyslog.d/50-default.conf and restart the service with $ service rsyslog restart.