Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have an 8 Gb log file (Rails production log). I need to cut it between some dates (lines). Which command could I use to do this?

share|improve this question
1  
Hey guys, this question is about a big file, so it's "Ante up!".. time matters...I've tested the favoured sed script on a real 8 GB file, with 85904064 lines (100 chars per line). I love sed, but as it stands, the sed script scans the entire file, every time. This makes it on average twice as slow as the awk script which exits-when-found... I think(?) the sed script may just need a q instead of d for the second expression... The test results are here: paste.ubuntu.com/573477 .. Also, it doesn't produce the proper output.. see my comment at the end asoundmove's answer. –  Peter.O Feb 28 '11 at 15:36
    
asoundmove's new sed version had addressed the speed issue, and it now matches awks's speed. and the new versin now outputs data correctly... see his comments for more detail. –  Peter.O Mar 1 '11 at 13:21
    
I just noticed you said "cut" (which typically means remove)... Do you really mean "cut", or do you mean "copy"? .... If you did mean "cut", then sed will do it easily. –  Peter.O Mar 5 '11 at 17:20

6 Answers 6

Something like

sed '1,/last date prior to chunk/d;/first date after chunk/,$d' logfile | tee cut-log | less

tee cut-log allows you to see on screen what is being put in file cut-log.

EDIT:

To satisfy fred.bear's exacting standards, here's a sed solution (though arguably the awk solution is a lot prettier):

b=BB; e=EE ;echo -e "AA\nAA\nBB\nBB\nCC\nCC\nDD\nDD\nEE\nEE\nFF\nFF" | sed -n ":b;/$b/b p;n;b b;:p;p;n;/$e/b e;b p;:e;p;n;/$e/b e;q"
share|improve this answer
    
-1 UUOC partmaps.org/era/unix/award.html –  dogbane Feb 25 '11 at 14:10
2  
@dogbane: yeah, yeah. Edited. I'm sure you sometimes write less than optimal code, does it deserve such a harsh comment? –  asoundmove Feb 25 '11 at 15:43
1  
note: If there are multiple consecutive 'first-date' lines with the same date, all but the first will not be deleted, and will be introduced to the output... just something to be aware of... (it depends on the situation) –  Peter.O Feb 25 '11 at 16:40
1  
... but, even though I a pro-sed ++, I think this particular job is beyond its limits, for anything other than one' own personal tool.. Here is the main issue sed has in this case (yours, and mine.. I managed to get sed to do the same as yours.. it also ran within 1%).. back to the main issue..(which does not apply to awk).... Bug(not fixable): Regarding a date which is valid within the scope of the log, but is not actually present in the log will, in the case of the 1st arg, cause sed to print nothing, and in the case of the 2nd arg, sed wil print everything after the first date! ... more... –  Peter.O Mar 1 '11 at 13:01
1  
Another, fixable bug: Is that it currently matches dates anywhewere in any line, including the data protion, but that's just a regex tweak.. And for anyone wanting to use it, perhaps you could comment that the args now refer to the first and last dates in range (not -1 and +1).. and finally.. my "exacting standards" are not mine. I am only the messenger of the Questioners request... The user will notice if it works as requested, or not.. This has been a great question for me.. I've learnt a lot :) ... and I glad to know that sed can match awk for speed, and it was actually a bit faster. –  Peter.O Mar 1 '11 at 13:13

If in your log file you have the dates in this format YYYY-MM-DD, then, to find all entries for say, 2011-02-10, you can do:

grep 2011-02-10 log_file

Now, say, if you want to find the entries for 2011-02-10 and 2011-02-11, then, again use grep but with multiple patterns:

grep -E '2011-02-10|2011-02-11' log_file
share|improve this answer
    
Good. It works "as advertised" :) ... However, grep will search the entire file, even if the date range is at the start of the file. On average this doubles the time of a search, when compared to "exit-after-last-item-in-range"... I'm only bothering to mention this because of the 8 GB file size mentioned in the question, Your grep time results are almost identical to the sed example here (1min 58sec). Here is the link to my time tests results: paste.ubuntu.com/573477 –  Peter.O Feb 28 '11 at 16:57

To print everything between FOO and BAR inclusive, try:

$ sed -n '/FOO/,/BAR/p' file.txt
share|improve this answer
1  
note: This will only print the first BAR of a series of consecutive BARS... –  Peter.O Feb 25 '11 at 16:35
    
another note... Big problem if either of the dates aren't present in the data.. If the last date is not present, sed will keep outputting lines until it reaches EOF. –  Peter.O Feb 26 '11 at 18:48

This will do what you want...
Both Including and Excluding the parameter dates are shown.

# set Test args
set  2011-02-24  2011-02-26  "junk"

from="$1"
till="$2"
file="$3"

# EITHER ====                              +++++++++  
# Ouptut lines between two parameter dates INCLUDING the parameter dates
  awk -v from=$from -v till=$till '
    ($2 >= from) && ($2 <= till) { print $0 ; next }
    ($2 > till) { exit }' "$file"

# OR ========                              ---------
# Ouptut lines between two parameter dates EXCLUDING the parameter dates
  awk -v from=$from -v till=$till '
    ($2 > from) && ($2 < till) { print $0 ; next }
    ($2 >= till) { exit }' "$file"

It tests for a (sorted) date in field 2... Here is an example fo the test data

    98  2011-02-05 xxxx
    99  2011-02-05 xxxx
   100  2011-02-06 xxxx
   101  2011-02-06 xxxx

And here is the test-data generator.

share|improve this answer
    
I would write it (taking the first one for example) a little more simply thus: awk -v from="$from" -v till="$till" '($2 >= from) { if ($2 <= till) { print } else { exit }' "$file" –  asoundmove Feb 28 '11 at 21:17
    
@asoundmove: Yes, that may look better, and it is definitely more conventional, but in reality, its execution time is only the duration of 1 extra if statement in total (not even 1 per line) ie. the logic flow is effectively the same, and the difference in run time would be counted in nanoseconds.... The only reason I didn't use "else" is that this is effectively my first ever awk script (aside from one day 4 years ago when I played with some examples)... and that is the first workable branch mechanism I found...(and as mentioned. it's just as fast).. I generly use sed Try q –  Peter.O Mar 1 '11 at 2:37
perl -wlne '/^2011-02-24/ .. /^2011-02-25/ and print' log_file
share|improve this answer
    
This will only print the first log entry for 2011-02-25 though. –  Gilles Feb 25 '11 at 19:54

Working with this size of files is always hard.

A way forward could be to split this file into a couple small ones, to do this you can use the split command.

split -d -l 50000 ToBigFile.data file_

Even thou it is split up you can still work with the file like if would be one using a bash for loop

for f in `ls file_*`; do cat $f; done;

But instead of the cat you can use inverted grep to get rid of unwanted data, that is irrelevant for this. (or the kind of refinement that you need).

At this point you will just work with a lot of smaller files, and the commands the others mentioned above will work smother on a lot of smaller files.

And when you are done, you can use a second for loop to build up the new smaller file again.

for f in `ls file_*`; do cat $f >> NewFile.data ; done;

Update Since we start to split the data in multiple files, there is going to be a lot of work with the harddrive and that takes time. (In this question apparently 5min).

On the other hand the next steps would probably be faster.

So this method is probably pointless for simple grep, awk, sed operation, but if the search patterns becomes more complicated it could become faster.

share|improve this answer
2  
Johanm, it takes awk and sed only 1 minute, on average, to search an 8 GB log file on my computer, and on the same compuer, just the inital file splitting, takes 4min 43sec ... :) –  Peter.O Mar 1 '11 at 13:49
    
Let's say that you could cut those awk and sed times by 50% on the smaller files. Then we still need to do more than 10 of those operation before we gain on the total time... So maybe the file split is not the best idea for a few regressions... –  Johan Mar 1 '11 at 15:03
    
The awk script could (easily) be modified to output 10 different search results to 10 files..in one single pass, but that would slow the read down while actually outputing the reports... Sed could also do the same, but as I've mentioned in asoundmove's comments, sed will fail if a particular date/time has no entry in the log(eg, you are searching by the hour).. I use sed a lot and it is extremely useful, but it has its limits... Here is a sed FAQ about when to use sed vs awk.. I don't necessarily agree with all of it, but I can see what they mean... sed.sourceforge.net/sedfaq6.html –  Peter.O Mar 2 '11 at 4:12

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.