I have an 8 Gb log file (Rails production log). I need to cut it between some dates (lines). Which command could I use to do this?

  • 1
    Hey guys, this question is about a big file, so it's "Ante up!".. time matters...I've tested the favoured sed script on a real 8 GB file, with 85904064 lines (100 chars per line). I love sed, but as it stands, the sed script scans the entire file, every time. This makes it on average twice as slow as the awk script which exits-when-found... I think(?) the sed script may just need a q instead of d for the second expression... The test results are here: paste.ubuntu.com/573477 .. Also, it doesn't produce the proper output.. see my comment at the end asoundmove's answer.
    – Peter.O
    Feb 28, 2011 at 15:36
  • asoundmove's new sed version had addressed the speed issue, and it now matches awks's speed. and the new versin now outputs data correctly... see his comments for more detail.
    – Peter.O
    Mar 1, 2011 at 13:21
  • I just noticed you said "cut" (which typically means remove)... Do you really mean "cut", or do you mean "copy"? .... If you did mean "cut", then sed will do it easily.
    – Peter.O
    Mar 5, 2011 at 17:20

6 Answers 6


Something like

sed '1,/last date prior to chunk/d;/first date after chunk/,$d' logfile | tee cut-log | less

tee cut-log allows you to see on screen what is being put in file cut-log.


To satisfy fred.bear's exacting standards, here's a sed solution (though arguably the awk solution is a lot prettier):

b=BB; e=EE ;echo -e "AA\nAA\nBB\nBB\nCC\nCC\nDD\nDD\nEE\nEE\nFF\nFF" | sed -n ":b;/$b/b p;n;b b;:p;p;n;/$e/b e;b p;:e;p;n;/$e/b e;q"
  • -1 UUOC partmaps.org/era/unix/award.html
    – dogbane
    Feb 25, 2011 at 14:10
  • 3
    @dogbane: yeah, yeah. Edited. I'm sure you sometimes write less than optimal code, does it deserve such a harsh comment?
    – asoundmove
    Feb 25, 2011 at 15:43
  • 1
    note: If there are multiple consecutive 'first-date' lines with the same date, all but the first will not be deleted, and will be introduced to the output... just something to be aware of... (it depends on the situation)
    – Peter.O
    Feb 25, 2011 at 16:40
  • 1
    ... but, even though I a pro-sed ++, I think this particular job is beyond its limits, for anything other than one' own personal tool.. Here is the main issue sed has in this case (yours, and mine.. I managed to get sed to do the same as yours.. it also ran within 1%).. back to the main issue..(which does not apply to awk).... Bug(not fixable): Regarding a date which is valid within the scope of the log, but is not actually present in the log will, in the case of the 1st arg, cause sed to print nothing, and in the case of the 2nd arg, sed wil print everything after the first date! ... more...
    – Peter.O
    Mar 1, 2011 at 13:01
  • 1
    Another, fixable bug: Is that it currently matches dates anywhewere in any line, including the data protion, but that's just a regex tweak.. And for anyone wanting to use it, perhaps you could comment that the args now refer to the first and last dates in range (not -1 and +1).. and finally.. my "exacting standards" are not mine. I am only the messenger of the Questioners request... The user will notice if it works as requested, or not.. This has been a great question for me.. I've learnt a lot :) ... and I glad to know that sed can match awk for speed, and it was actually a bit faster.
    – Peter.O
    Mar 1, 2011 at 13:13

To print everything between FOO and BAR inclusive, try:

$ sed -n '/FOO/,/BAR/p' file.txt
  • 1
    note: This will only print the first BAR of a series of consecutive BARS...
    – Peter.O
    Feb 25, 2011 at 16:35
  • another note... Big problem if either of the dates aren't present in the data.. If the last date is not present, sed will keep outputting lines until it reaches EOF.
    – Peter.O
    Feb 26, 2011 at 18:48

This will do what you want...
Both Including and Excluding the parameter dates are shown.

# set Test args
set  2011-02-24  2011-02-26  "junk"


# EITHER ====                              +++++++++  
# Ouptut lines between two parameter dates INCLUDING the parameter dates
  awk -v from=$from -v till=$till '
    ($2 >= from) && ($2 <= till) { print $0 ; next }
    ($2 > till) { exit }' "$file"

# OR ========                              ---------
# Ouptut lines between two parameter dates EXCLUDING the parameter dates
  awk -v from=$from -v till=$till '
    ($2 > from) && ($2 < till) { print $0 ; next }
    ($2 >= till) { exit }' "$file"

It tests for a (sorted) date in field 2... Here is an example fo the test data

    98  2011-02-05 xxxx
    99  2011-02-05 xxxx
   100  2011-02-06 xxxx
   101  2011-02-06 xxxx

And here is the test-data generator.

  • I would write it (taking the first one for example) a little more simply thus: awk -v from="$from" -v till="$till" '($2 >= from) { if ($2 <= till) { print } else { exit }' "$file"
    – asoundmove
    Feb 28, 2011 at 21:17
  • @asoundmove: Yes, that may look better, and it is definitely more conventional, but in reality, its execution time is only the duration of 1 extra if statement in total (not even 1 per line) ie. the logic flow is effectively the same, and the difference in run time would be counted in nanoseconds.... The only reason I didn't use "else" is that this is effectively my first ever awk script (aside from one day 4 years ago when I played with some examples)... and that is the first workable branch mechanism I found...(and as mentioned. it's just as fast).. I generly use sed Try q
    – Peter.O
    Mar 1, 2011 at 2:37
  • I don't understand where you give the text file name and location in this method? can someone help me see through my stupidity
    – Giles
    Jul 11, 2016 at 12:14

If in your log file you have the dates in this format YYYY-MM-DD, then, to find all entries for say, 2011-02-10, you can do:

grep 2011-02-10 log_file

Now, say, if you want to find the entries for 2011-02-10 and 2011-02-11, then, again use grep but with multiple patterns:

grep -E '2011-02-10|2011-02-11' log_file
  • Good. It works "as advertised" :) ... However, grep will search the entire file, even if the date range is at the start of the file. On average this doubles the time of a search, when compared to "exit-after-last-item-in-range"... I'm only bothering to mention this because of the 8 GB file size mentioned in the question, Your grep time results are almost identical to the sed example here (1min 58sec). Here is the link to my time tests results: paste.ubuntu.com/573477
    – Peter.O
    Feb 28, 2011 at 16:57

Working with this size of files is always hard.

A way forward could be to split this file into a couple small ones, to do this you can use the split command.

split -d -l 50000 ToBigFile.data file_

Even thou it is split up you can still work with the file like if would be one using a bash for loop

for f in `ls file_*`; do cat $f; done;

But instead of the cat you can use inverted grep to get rid of unwanted data, that is irrelevant for this. (or the kind of refinement that you need).

At this point you will just work with a lot of smaller files, and the commands the others mentioned above will work smother on a lot of smaller files.

And when you are done, you can use a second for loop to build up the new smaller file again.

for f in `ls file_*`; do cat $f >> NewFile.data ; done;

Update Since we start to split the data in multiple files, there is going to be a lot of work with the harddrive and that takes time. (In this question apparently 5min).

On the other hand the next steps would probably be faster.

So this method is probably pointless for simple grep, awk, sed operation, but if the search patterns becomes more complicated it could become faster.

  • 3
    Johanm, it takes awk and sed only 1 minute, on average, to search an 8 GB log file on my computer, and on the same compuer, just the inital file splitting, takes 4min 43sec ... :)
    – Peter.O
    Mar 1, 2011 at 13:49
  • Let's say that you could cut those awk and sed times by 50% on the smaller files. Then we still need to do more than 10 of those operation before we gain on the total time... So maybe the file split is not the best idea for a few regressions...
    – Johan
    Mar 1, 2011 at 15:03
  • The awk script could (easily) be modified to output 10 different search results to 10 files..in one single pass, but that would slow the read down while actually outputing the reports... Sed could also do the same, but as I've mentioned in asoundmove's comments, sed will fail if a particular date/time has no entry in the log(eg, you are searching by the hour).. I use sed a lot and it is extremely useful, but it has its limits... Here is a sed FAQ about when to use sed vs awk.. I don't necessarily agree with all of it, but I can see what they mean... sed.sourceforge.net/sedfaq6.html
    – Peter.O
    Mar 2, 2011 at 4:12
perl -wlne '/^2011-02-24/ .. /^2011-02-25/ and print' log_file

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy