How to cut part from log file?

Question

I have 8 Gb long log file (Rails production log), and I need to cut part between some dates (lines). Which command i have to use, to do this?

Answer 1

Something like

sed '1,/last date prior to chunk/d;/first date after chunk/,$d' logfile | tee cut-log | less

tee cut-log allows you to see on screen what is being put in file cut-log.

EDIT:

To satisfy fred.bear's exacting standards, here's a sed solution (though arguably the awk solution is a lot prettier):

b=BB; e=EE ;echo -e "AA\nAA\nBB\nBB\nCC\nCC\nDD\nDD\nEE\nEE\nFF\nFF" | sed -n ":b;/$b/b p;n;b b;:p;p;n;/$e/b e;b p;:e;p;n;/$e/b e;q"

Answer 2

If in your log file you have the dates in this format YYYY-MM-DD, then, to find all entries for say, 2011-02-10, you can do:

grep 2011-02-10 log_file

Now, say, if you want to find the entries for 2011-02-10 and 2011-02-11, then, again use grep but with multiple patterns:

grep -E '2011-02-10|2011-02-11' log_file

Answer 3

To print everything between FOO and BAR inclusive, try:

$ sed -n '/FOO/,/BAR/p' file.txt

Answer 4

This will do what you want...
Both Including and Excluding the parameter dates are shown.

# set Test args
set  2011-02-24  2011-02-26  "junk"

from="$1"
till="$2"
file="$3"

# EITHER ====                              +++++++++  
# Ouptut lines between two parameter dates INCLUDING the parameter dates
  awk -v from=$from -v till=$till '
    ($2 >= from) && ($2 <= till) { print $0 ; next }
    ($2 > till) { exit }' "$file"

# OR ========                              ---------
# Ouptut lines between two parameter dates EXCLUDING the parameter dates
  awk -v from=$from -v till=$till '
    ($2 > from) && ($2 < till) { print $0 ; next }
    ($2 >= till) { exit }' "$file"

---

It tests for a (sorted) date in field 2... Here is an example fo the test data

    98  2011-02-05 xxxx
    99  2011-02-05 xxxx
   100  2011-02-06 xxxx
   101  2011-02-06 xxxx

---

And here is the test-data generator.

Answer 5

perl -wlne '/^2011-02-24/ .. /^2011-02-25/ and print' log_file

Answer 6

Working with this size of files is always hard.

A way forward could be to split this file into a couple small ones, to do this you can use the split command.

split -d -l 50000 ToBigFile.data file_

Even thou it is split up you can still work with the file like if would be one using a bash for loop

for f in `ls file_*`; do cat $f; done;

But instead of the cat you can use inverted grep to get rid of unwanted data, that is irrelevant for this. (or the kind of refinement that you need).

At this point you will just work with a lot of smaller files, and the commands the others mentioned above will work smother on a lot of smaller files.

And when you are done, you can use a second for loop to build up the new smaller file again.

for f in `ls file_*`; do cat $f >> NewFile.data ; done;

---

Update Since we start to split the data in multiple files, there is going to be a lot of work with the harddrive and that takes time. (In this question apparently 5min).

On the other hand the next steps would probably be faster.

So this method is probably pointless for simple grep, awk, sed operation, but if the search patterns becomes more complicated it could become faster.