How to track the number of processes and forks per user?

Question

I need to figure out how many forks are done and how many concurrent processes are run by each user over time. It does not look like this information is tracked by my distribution.

I know how to sets limits, but I'm interested in tracking these numbers for each user.

Answer 1

Try the psacct package (GNU accounting), it should do just about everything you need, once installed and enabled (accton), then lastcomm will keep report on user processes (see also sa and dump-acct). See this for reference: User's executed commands log file

You might need to upgrade the version to log PID/PPID, see http://serverfault.com/questions/334547/how-can-i-enable-pid-and-ppid-fields-in-psacct-dump-acct , otherwise I suspect it will under-report on fork() without exec().

Update If your lastcomm outputs F in the 2nd column it means the process was a fork (that never called exec() to replace itself with a new process). The output of dump-acct should show you the PID (and PPID) in acct v3 format.

An alternative to psacct might be the new(ish) taskstats, there's not a huge amount of support for it yet AFAICT, see Documentation/accounting/taskstats.txt in your kernel version source. This might help get you started http://code.google.com/p/arsenalsuite/wiki/TrackingIOUsage and there's a perl module Linux::Taskstats::Read on CPAN.

In either case you'll need to process the data based on timestamps if you want the concurrent process count per-user.

Update 2 Ok, the things to check for the required psacct support are:

1. (official) kernel >= 2.6.8 for v3 accounting support (or backport)
2. kernel with CONFIG_BSD_PROCESS_ACCT and CONFIG_BSD_PROCESS_ACCT_V3 enabled
3. v3 capable accounting (psacct) package, as noted above

All of the above should be true in CentOS 6, I've checked a 5.x and it does not have CONFIG_BSD_PROCESS_ACCT_V3=y, so you would have to rebuild your kernel to enable it.

The original psacct-6.3.2 is about 15 years old, the Red Hat/CentOS version has backported v3 and PID display support (I can't test it right now, but it should work).

To check a your kernel config:

zgrep BSD_PROCESS_ACCT /proc/config.gz /boot/config-`uname -r`

Answer 2

You can use ps -ef to look at the child pid, the parent id is the one that did the forking or spawning. You could use watch to run a script on a regular basis and do the counting for you.

Answer 3

You can set up the audit subsystem to log forks (and whatever other system call catches your fancy).

auditctl -a exit,always -S clone