1

I have a monitoring server (‘ClusterControl’) which uses a regular user account to login using an ‘interactive shell’ and runs some sudo maintenance commands.

This account logs-in via an SSH Key (passwordless) 4 times per minute it seems.

The /var/log/wtmp, /var/log/auth.log and /var/log/syslog fill up with thousands of login and sudo entries.

Three problems arise:

  1. The last command without parameters is rendered un-usable since the serviceaccount username is displayed thousands of times and scrolls of the screen - other account logins are not seen.

  2. The files /var/log/wtmp, /var/log/auth.log and /var/log/syslog get very large in size very quickly.

  3. Syslog at /var/log/syslog is un-usable without passing it through grep to filter out the login entries.

Question

How can I exclude this user account from being logged?

P.S. I have tried adding a PAM entry of systemd [success=ok default=1] pam_succeed_if.so user in username in the various pam files located in /etc/pam.d folder as some articles have directed but it has not worked.

I’m not sure which file to use and what the correct syntax should be.

/var/log/syslog shows the ssh username login logged by ‘systemd’ at the beginning of the log line.

The server is Ubuntu 18.04

EDIT

ClusterControl runs an interactive shell using ssh -t when it logs in as stated by one of the company’s System Support Engineers in a blog post here. However, their suggestion is merly to enable more frequent log rotations.

5
  • You need to give more information on how you connect and execute commands via ssh. Generally, when you execute a command like ssh server command, no login shell is executed on the server, therefore nothing is recorded into /var/log/wtmp.
    – xhienne
    Mar 2 at 23:34
  • @xhienne I only know that ClusterControl runs an interactive shell using ssh -t when it logs in as stated by one of the company’s System Support Engineers in a blog post here. However, their suggestion is merly to enable more frequent log rotation. Mar 3 at 2:56
  • I don't know ClusterControl but the way it works is the root of your problems. As I said, ssh server command doesn't log in wtmp. But they probably chose to echo command | ssh -t server or something similar, so that they can use sudo. Remove the -t and your problem is gone. IMO, your question is specific to this commercial product ClusterControl.
    – xhienne
    Mar 3 at 9:58
  • @xhienne 2 things: 1) Am I to understand that it is possible for the product developers to remove the ssh -t method and still use sudo? 2) In the absence of a solution from the product developers, my question still stands. How can I exclude this and only this user from being logged to wtmp? There must be a way I believe. Mar 3 at 11:14
  • 2
    1. Quoting the page you linked to: "On Debian/Ubuntu system, sudo user does not need to acquire tty as it defaults to have no “requiretty” configured. However, ClusterControl defaults to append -t flag if it detects the SSH user as a non-root user." => "ClusterControl defaults to" suggests this can be disabled. 2. Yes, your question is legitimate and has probably a solution on the server side, that's why it is still opened, waiting for someone to answer ;-)
    – xhienne
    Mar 3 at 11:20
2

There are 2 choices.

TL;DR

1. Syslog Filters

Using rsyslog filters is by far the quickest solution but not as comprehensive as PAM.

At the top of your rsyslog default conf file, add the keywords you wish to filter on and the filter command. In this case, the filter command is delete ('~').

$ vi /etc/rsyslog.d/50-default.conf
:msg, contains, "clustercontrol" ~
:msg, contains, "pam_unix(sudo:session): session closed for user root" ~
:msg, contains, "Removed session" ~

2. Manage Log Files

While I managed to stop thousands of logs being recorded per day in /var/log/syslog & /var/log/auth.log there didn't seem to be a way to stop logging the service account SSH logins from clustercontrol to the /var/log/wtmp and /var/log/lastlog files.

For this, we use logrotate (See below).

Using PAM

PAM Debug

The first step to understanding what was going on was to enable debug messages. The problem was that on my Ubuntu 18.04, the pam debug messages were not being logged - until I explicitly told rsyslog which file to record debug messages in using:

$ vi /etc/rsyslog.d/50-default.conf
*.debug           /var/log/debug.log

$ service rsyslog restart

pam_succeed_if.so

This is the pam module we will use to write conditional tests using pam. The documentation is a little in-complete, dry and not very verbose on just how the rules work and what all the directives do. After reading a dozen or so articles, here's what I managed to piece together:

Understand who's doing the logging

Inside /var/log/auth.log you'll see the log messages we wish to stop for this user account:

Mar 17 13:05:27 dev1 sshd[18833]: pam_unix(sshd:session): session opened for user clustercontrol by (uid=0)

Mar 17 13:04:25 dev1 sudo: pam_unix(sudo:session): session opened for user root by clustercontrol(uid=0)

From the above logs we can tell that pam_unix.so is the pam authentication module doing the logging. It's being called by two separate services sshd and sudo.

The pam type token tells pam what type of authentication is to be used for this module. In this case, session which are things that should be done before and/or after the user is authenticated.

Responsible PAM Config

So lets find the pam module pam_unix.so responsible for these logs in our system's configuration.

$ cd /etc/pam.d

$ grep 'pam_unix.so' * -in
common-account:17:account       [success=1 new_authtok_reqd=done default=ignore]        pam_unix.so 
common-auth:17:auth     [success=1 default=ignore]      pam_unix.so nullok_secure
common-password:25:password     [success=1 default=ignore]      pam_unix.so obscure sha512
common-session:29:session       required        pam_unix.so 
common-session-noninteractive:30:session        required        pam_unix.so 
runuser:5:session               required        pam_unix.so

From above, we can see that pam_unix.so module for ':session' is in the file common-session on line #29.

IMPORTANT
  1. Find the line in /etc/pam.d/sshd that includes the common-session file and add your custom pam_succeed_if.so statement just before it.

  2. Now do almost exactly the same for the /etc/pam.d/sudo file, but here our custom rule syntax is a little different.

Custom PAM Rules

Sequence is Important

Just before our pam_unix.so rules are run from the @include common-session file, we place our pam_succeed.so module rule.

sshd rule
$ grep -in include /etc/pam.d/sshd
5:@include common-auth
15:@include common-account
29:@include common-session
32:# This includes a dynamically generated part from /run/motd.dynamic
56:@include common-password

Line 29 in our case.

$ vi /etc/pam.d/sshd +29
session [success=done default=ignore] pam_succeed_if.so quiet service in sshd user = clustercontrol
# Standard Un*x session setup and teardown.
@include common-session

Its easier if we read the rules from back to front:

We check if a user named clustercontrol (user = clustercontrol) is authenticated by sshd (service in sshd) then this rule matches and we run the directive (quiet) which makes logging quiet.

The part [success=done means, if you get a match to this rule, we're done so skip processing the next rules, or skip over the next rule in the pam stack.

This effectively stops sshd logging for this user.

sudo rule

For this I added line #6 to /etc/pam.d/sudo :

$ vi /etc/pam.d/sudo
:set number 

  1 #%PAM-1.0
  2 
  3 session    required   pam_env.so readenv=1 user_readenv=0
  4 session    required   pam_env.so readenv=1 envfile=/etc/default/locale user_readenv=0
  5 
  6 session [success=done default=ignore] pam_succeed_if.so quiet uid = 0 ruser = clustercontrol
  7 @include common-auth
  8 @include common-account
  9 @include common-session-noninteractive

session [success=done default=ignore] pam_succeed_if.so quiet uid = 0 ruser = clustercontrol

Our syntax is a little different here. Sudo generally runs commands as the root user (unless special directives tell it otherwise). On unix like systems that's user 0.

So our custom rule is saying, if the sudo command is run in the context of user with uid 0 by the remote user (ruser), in our case this is clustercontrol, then be quiet with logging and we're done so skip any further rules for this user of type session authentication.

NB
Don't forget to comment out the *.debug line in /etc/rsyslog.d/50-default.conf and restart the service with $ service rsyslog restart.

Logrotate

Lastlog + wtmp

Even setting noupdate or nowtmp directives for pam_lastlog.so globally, had no effect on my system so I simply made sure to compress and rotate these logs regularly using the logrotate config below.

$ vi /etc/logrotate.conf
# no packages own wtmp, or btmp -- we'll rotate them here
/var/log/wtmp {
    missingok
    monthly
    create 0664 root utmp
    rotate 12
    size 200M
    compress
}

/var/log/btmp {
    missingok
    monthly
    create 0660 root utmp
    rotate 12
    size 200M
    compress
}

This will rotate the logs monthly or every 200Mb, whichever comes first. Logs will be compressed and retained for a maximum of 12 months.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.