Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I often have longer redundant log files, redundant in the sense that blocks of text repeat with minor changes. The length of the files makes it difficult to spot the actual information. I had the idea to write a script which identifies recurrences and reduces the file by replacing each recurrence by the output of diff for consecutive blocks. Before I start I would like to know if such a tool (or maybe option to diff) already exists.

share|improve this question

You'll probably like to have a look at user Drench approach to a similar problem, in an answer he gave on this very site : http://unix.stackexchange.com/a/837/27616 (ie, using git. Allowing to store just the diffs, retrieve any version in time, etc. And he gives a link to a blog with additionnal infos about pros/cons/caveats)

share|improve this answer
Can this be used without git? – highsciguy Feb 6 '13 at 10:32

You mostly search in logfiles, and cutting out information that might be needed isn't a good idea. That said, there are tools like logrotate, which rotates the log files after some time (say monthly), and asking Google on "unix logfile searching" gives some 5 million hits (yes, this is a common task).

share|improve this answer
Well, I am talking about files which are really redundant and the 'compression' I am thinking of should be loss-less. Also the log files are not standard unix log files. They are rather produced by my own code. I would like to post-process them using unix tools such as diff to simplify them for a human reader. – highsciguy Feb 11 '13 at 8:55

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.