1

I am attempting to find the best way to determine when the second file (of a matching criteria) is created. The context is an audit log rotation.

Given a directory where audit logs are created every hour, I need to execute a parsing awk script upon the audit log that has been closed off. What I mean by that, is that every hour a new audit log is created, and the old audit log is closed, containing up to an hours worth of information. The new log file is to be left alone until it too is closed and a new one created.

I could create a bash shell script and use find /home/tomcat/openam/openam/log -name amAuthentication.* -mmin -60 and then have this executed every 10 minutes via a crontab entry but I'm not sure how to write the rest of it.

I suppose the script could start off by saving the contents of that find to a temp file, and upon every crontab execution, compare the new find command contents and when it changes, use the temp file as the input to the awk script. When the awk script is complete, save the contents of the new find to the file.

A colleague has suggested I use the old school 'sticky-bit' to flag processed files. Perhaps this is the way to proceed.

2
  • Perhaps this should be on stack overflow?
    – D-Klotz
    Jul 21, 2014 at 19:06
  • Bash isn't required, it happens to be what I'm familiar with. If perl or python can provide a better solution, I welcome the answer.
    – D-Klotz
    Jul 21, 2014 at 19:20

2 Answers 2

Reset to default
0

Try to use inotifywait for that:

inotifywait -e close_write /home/tomcat/openam/openam/log/CURRENT_OPENED_LOG_FILE
3
  • Unfortunately, these production systems do not have that package. Some are as old as this: SUSE Linux Enterprise Server 11 (x86_64) VERSION = 11 PATCHLEVEL = 0
    – D-Klotz
    Jul 21, 2014 at 20:09
  • @D-Klotz If your logs rotating every hour, just find the script for rotating and add your own script to the end. Or add a new line to crontab. Something like 10 * * * * your script
    – dchirikov
    Jul 21, 2014 at 20:13
  • The logs are rotated by OpenAM. I'd like to avoid that idea if at all possible. Upgrades are fun enough as it is.
    – D-Klotz
    Jul 21, 2014 at 20:25
0

For closure, this is the start of the script I am going to use. It needs more work to make it robust and do logging but you should get the general idea. 

#!/bin/sh

# This script should be executed from a crontab that executes every 5 or 10 minutes
# the find below looks for all log files that do NOT have the sticky bit set. 
# You can see the sticky bit with a "T" from a "ls -l". 
for x in `find /home/tomcat/openam/openam/log/ -name "log-*" -type f ! -perm -1000 -print`
do
    # Look for open files. For safety, log that we are skipping them
    if lsof | grep $x > /dev/null; then
        # create a log entry on why I'm not processing this file...
        echo $x " is open"
    else 
        # $x "is closed and not sticky"
        # run the awk scripts to process the file!
        echo $x " processing with awk..." 
        # Set the sticky bit to indicate we have processed this file
        chmod +t $x
    fi
done
1
  • You might find fuser faster and easier, fuser -s filename will (silently) return 0 if filename is open. For similar purposes I usually use a program called retail which cashes filename/size/inode so log content can be incrementally processed without missing anything. Whether you can use that depends on the filename(s) and rotation mechanism (i.e. preservation of inode number).
    – mr.spuratic
    Jul 24, 2014 at 9:23

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .