I had a outage incident a few days ago and the support team has asked for any available logs for the period, while thinking about how to do this, it occurred to me that it would be generally handy to be able to parse many log files at once for entries between 2 dates.

I see many tools that would make this possible, had we had the foresight to configure them, not least;
splunk, syslog-ng, rsyslog, and others that stash logs to some searchable backend such as mysql.

(I am definitely planning to install a central log host to fulfill this requirement)

However for the short term I am interested in a tool that would blaze through the /var/log/* folder and dump to stdout, preferably with a simple options syntax (find options are nice +mtime etc) and written in something that can do this fast (so probably C rather than perl or ruby)

I see a bunch of tools that I am looking at which may or may not be fit for purpose, such as mergelog, logpp, sec, but none are really jumping out as obviously good for this, which is odd because I would expect this to be the sort of thing that is done to death on unix tools.

  1. Use:

    find ... -mtime [fromdate] -a -not -ctime [todate]

    to find all files modified after [fromdate] and created before [todate]

  2. For found files invent a creative pattern to extract lines of interest. Use egrep/awk as they are much faster than perl/python. Example:

    awk '/^2012-01-(01|02|03)/'

    would print all lines for first, second, and third of January 2012.

Above is 2 minutes to write and run. If the performance proves really an issue, then move on and try finding something more complicated.

| improve this answer | |
  • yeah, a bash loop and sed did the trick for the initial purpose. – Tom H Feb 24 '12 at 15:24
  • however the date format in these syslog files is retarded, it doesn't even have a year Feb 22 06:57:56 workstation001 kernel: [ 0.124272] ACPI: Interpreter enabled etc – Tom H Feb 24 '12 at 15:37
  • is [ 0.124272] the time since the box start? – bobah Feb 24 '12 at 16:13
  • err, not sure, seems like that. i just picked some random entry from messages on an example box. I guess if someone still has log entries for Feb 22 2011, then they are probably not looking after the logrotate properly, but its annoying to have to guess the year. – Tom H Feb 24 '12 at 16:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.