I need to process a very large log file with many lines in different formats.
My goal is to extract unique line entries who have the same starting pattern, e.g. '^2011-02-21.*MyKeyword.*Error', effectively obtaining a list of samples for each line pattern, therefore identifying the patterns.
I only know a few patterns so far, and browsing through the file manually is definitely not the option.
Please note that besides the known patterns, there is a number of unknown ones too, and I'd like to automate extracting those as well.
What is the best way to do this? I do know regular expressions quite well, but haven't done much work with awk/sed which I imagine would be used at some point in this process.