This was a very time consuming task. And yes, it certainly needed to be scripted if you're going to search for multiple strings in multiple different logs at the same time. But I recently had to do this and it was quite painful. Nevertheless, it is done and ready and can be downloaded from the following link:
Log Search Script Download
The way this works is pretty simple.
Scenario 1: Monitor ONE string in just ONE log file
./logsearch.sh localhost /var/tmp/logXray autonda /var/log/messages 60m 'can.*t.*open' '.' 1 2 single_errCheck -ndshow
Scenario 2: Monitor MULTIPLE strings in just ONE log file
./logsearch.sh localhost /var/tmp/logXray autonda /var/log/messages 60m 'can.*t.*open_P_ntpd.*stat' '.' 1 2 multi_errCheck -ndshow
Scenario 3: Monitor Single/Multiple strings in Multiple log files
./logsearch.sh localhost /var/tmp/logXray autonda /var/log 60m 'can.*t.*open_P_ntpd.*stat' '.' 1 2 multi_err_multi_logCheck -ndshow
The _P_ means OR - It replaces the pipe "|" symbol because it is less likely you'll have to search for a string containing "_P_". If you dont wish to type "_P_", you can just substitute the _P_ with "|".
When using this script, the parameters you'll be changing frequently are:
- The log file or log directory to be monitored
- The age a log file must be for it to be monitored..i.e. do not monitor or discover any log file that has a timestamp over 60 minutes
- The strings(s)/pattern(s) you want to watch for
- The tag - this is the second to last argument you have to supply. It records stats about the log file(s) you're monitoring under /var/tmp/logXray
- The log option -ndshow - This is the parameter you want to use if you wish to output the entries from the logs found matching the pattern(s) you specified. If you just want to see the total count of each pattern found, simply replace '-ndshow' with '-ndfoundmul'.
When using '-ndfoundmul', you'll get an output similar to:
[root@dgphxtest001]# ./logsearch.sh localhost /var/tmp/logXray autonda /var/log/messages 60m 'can.*t.*open_P_ntpd.*stat' '.' 1 2 blahblahA -ndfoundmul
OK: [/var/log/messages] /var/log/messages:P=(can_t_open=0 ntpd_stat=0)_F=(117s)_R=(228,228=0)
Solution to the Original Poster's Issue: Scan for Multiple strings in multiple log files
./logsearch.sh localhost /var/tmp/logXray autonda /var/log 60m 'fatal_P_error_P_critical_P_failure_P_warning' '.' 1 2 multierr_logCheck -ndshow
OSes: This was tested on Ubuntu and Red Hat