Reduce redundant logfile

Question

I often have longer redundant log files, redundant in the sense that blocks of text repeat with minor changes. The length of the files makes it difficult to spot the actual information. I had the idea to write a script which identifies recurrences and reduces the file by replacing each recurrence by the output of diff for consecutive blocks. Before I start I would like to know if such a tool (or maybe option to diff) already exists.

Answer 1

You'll probably like to have a look at user Drench approach to a similar problem, in an answer he gave on this very site : http://unix.stackexchange.com/a/837/27616 (ie, using git. Allowing to store just the diffs, retrieve any version in time, etc. And he gives a link to a blog with additionnal infos about pros/cons/caveats)

Answer 2

You mostly search in logfiles, and cutting out information that might be needed isn't a good idea. That said, there are tools like logrotate, which rotates the log files after some time (say monthly), and asking Google on "unix logfile searching" gives some 5 million hits (yes, this is a common task).