I am trying to find how to log a specific instantiation of rrdtool to see whether the path it is receiving is incorrect.

I know I could wrap the executable in a shell script that would log the parameters, but I was wondering if there was a more kernel-specific way to monitor for that, perhaps a filesystem callback that sees when a particular /proc/pid/exe matches a given binary?


Yes, there is a kernel facility: the audit subsystem. The auditd daemon does the logging, and the command auditctl sets up the logging rules. You can log all calls to a specific system alls, with some filtering. If you want to log all commands executed and their arguments, log the execve system call:

auditctl -a exit,always -S execve

To specifically trace the invocation of a specific program, add a filter on the program executable:

auditctl -a exit,always -S execve -F path=/usr/bin/rrdtool

The logs show up in /var/log/audit.log, or wherever your distribution puts them. You need to be root to control the audit subsystem.

Once you're done investigating, use the same command line with -d instead of -a to delete a logging rule, or run auditctl -D to delete all audit rules.

For debugging purposes, replacing the program by a wrapper script gives you more flexibility to log things like the environment, information about the parent process, etc.

  • Why the -F path=/ust/bin/rrdtool? I don't get how rrdtool is even related software. – Graeme Feb 27 '14 at 12:28
  • @Graeme The problem described in the question was tracking an invocation of rrdtool. If you want to log invocations of all programs, drop the -F path=… part (you'll get a lot of logs of course). – Gilles 'SO- stop being evil' Feb 27 '14 at 12:37
  • Right... first line of the question. Thanks. – Graeme Feb 27 '14 at 12:40
  • This is all great, but how do I reset the config it to initial state? Otherwise it'll continue filling the log with new and new launched commands... or is this auditctl command only effective until reboot? – Ruslan May 25 '19 at 16:19
  • @Ruslan The effect of auditctl only survives until reboot, but that's a good point anyway, I've added instructions on removing them without rebooting to my answer. – Gilles 'SO- stop being evil' May 25 '19 at 16:25

You could use snoopy.

Snoopy is more lightweight solution as it does not need kernel cooperation. All that is needed is dynamic loader (dl) that preloads snoopy library, path to which is specified in /etc/ld.so.preload.

Disclosure: I am current snoopy maintainer.

  • Is it possible to do logging for commands directly or indirectly spawned from a particular shell only? – r.v Apr 10 '15 at 21:03
  • I am not sure I understand your question - do you mean "shell" as specific program that is used as shell (bash, dash, zsh etc.), or you mean you would like to log just specific PTY? Snoopy provides filtering framework, but currently only a couple of very basic filters are implemented, see here for the list: link. If you have a concrete usecase that might be applicable to others, please do explain in feature request, and, oh, btw, patches are welcome :) – Bostjan Skufca Apr 15 '15 at 16:33
  • I meant just a specific PTY. – r.v Apr 15 '15 at 16:46
  • There is no specific filter for PTY available ATM. However, you could use snoopy to log everything, including which PTY the event occured on, and then do the filtering in your syslog daemon. I do not know which one you are using, but syslog-ng (for example) can do regex matching, positive or negative. – Bostjan Skufca Apr 15 '15 at 17:12
  • Sure, thanks! The tool and the approach is in general quite useful. I can easily do some filtering to get what I need. – r.v Apr 15 '15 at 17:41

The Linux kernel "audit" subsystem can do what you need.

e.g. if you run these commands:

auditctl -a exit,always -F arch=b64 -S execve
auditctl -a exit,always -F arch=b32 -S execve

Then every execution event is logged, and a lot of information is provided around that

e.g. this is the output of me running tail /var/log/audit/audit.log

exit=0 a0=7f0e4a21e987 a1=7f0e4a21e6b0 a2=7f0e4a21e808 a3=8 items=2 ppid=906 pid=928 auid=500 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=1 comm="tail" exe="/usr/bin/tail" subj=kernel key=(null)
type=EXECVE msg=audit(1543671660.203:64): argc=2 a0="tail" a1="/var/log/audit/audit.log"
type=CWD msg=audit(1543671660.203:64):  cwd="/home/sweh"
type=PATH msg=audit(1543671660.203:64): item=0 name="/usr/bin/tail" inode=266003 dev=fd:03 mode=0100755 ouid=0 ogid=0 rdev=00:00 obj=unlabeled objtype=NORMAL cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0
type=PATH msg=audit(1543671660.203:64): item=1 name="/lib64/ld-linux-x86-64.so.2" inode=273793 dev=fd:03 mode=0100755 ouid=0 ogid=0 rdev=00:00 obj=unlabeled objtype=NORMAL cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0
type=PROCTITLE msg=audit(1543671660.203:64): proctitle=7461696C002F7661722F6C6F672F61756469742F61756469742E6C6F67

There's some interesting values that can be seen; e.g. "auid" is 500, which is my login ID, even though "uid" is zero ('cos I'm running under su). So even though the user may have switched accounts with su or sudo we can still track back to their "audit ID"

Now those auditctl commands will be lost on a reboot. You can put them into a configuration file (eg in the /etc/audit/rules.d/ directory, on CentOS 7). The exact location will depend on your OS version. The auditctl manual page should help here.

Beware, though... this will cause a lot of log messages to be generated. Make sure you have enough space on the disk!

If necessary the rules can be limited to a specific user, or a specific command.

And also beware; if a user puts the password in the command execution (e.g. mysql --user=username --password=passwd) then this will be logged.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.