I have 8 Gb long log file (Rails production log), and I need to cut part between some dates (lines). Which command i have to use, to do this?
To satisfy fred.bear's exacting standards, here's a sed solution (though arguably the awk solution is a lot prettier):
|show 7 more comments|
If in your log file you have the dates in this format
Now, say, if you want to find the entries for 2011-02-10 and 2011-02-11, then, again use
To print everything between FOO and BAR inclusive, try:
This will do what you want...
It tests for a (sorted) date in field 2... Here is an example fo the test data
And here is the test-data generator.
Working with this size of files is always hard.
A way forward could be to split this file into a couple small ones, to do this you can use the split command.
Even thou it is split up you can still work with the file like if would be one using a bash for loop
But instead of the cat you can use inverted grep to get rid of unwanted data, that is irrelevant for this. (or the kind of refinement that you need).
At this point you will just work with a lot of smaller files, and the commands the others mentioned above will work smother on a lot of smaller files.
And when you are done, you can use a second for loop to build up the new smaller file again.
Update Since we start to split the data in multiple files, there is going to be a lot of work with the harddrive and that takes time. (In this question apparently 5min).
On the other hand the next steps would probably be faster.
So this method is probably pointless for simple grep, awk, sed operation, but if the search patterns becomes more complicated it could become faster.