Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

Let's say, I have a really big text file (about 10.000.000 lines). I need to grep it from the end and save result to a file. What's the most efficient way to accomplish task?

share|improve this question
8  
Use tac and grep to achieve what you want. –  val0x00ff Jul 23 at 12:06
    
In addition to the excellent solutions posted, GNU grep has a --max-count (number) switch that aborts after a certain number of matches, which might be interesting to you. –  Ulrich Schwarz Jul 23 at 13:28
    
@val0x00ff could you take a look at this question –  c0rp Jul 24 at 8:50

4 Answers 4

tac/grep Solution

tac file | grep whatever

Or a bit more effective:

grep whatever < <(tac file)

Time with a 500MB file:

real    0m1.225s
user    0m1.164s
sys     0m0.516s

sed/grep Solution:

sed '1!G;h;$!d' | grep whatever

Time with a 500MB file: Aborted after 10+ minutes.

awk/grep Solution:

awk '{x[NR]=$0}END{while (NR) print x[NR--]}' file | grep whatever

Time with a 500MB file:

real    0m5.626s
user    0m4.964s
sys     0m1.420s

perl/grep Solution:

perl -e 'print reverse <>' file | grep whatever

Time with a 500MB file:

real    0m3.551s
user    0m3.104s
sys     0m1.036s
share|improve this answer
2  
sed, awk and perl (with this method) are not OK since they read the file from the beginning, which is very inefficient. I suppose that tac does the right thing. –  vinc17 Jul 23 at 12:39
1  
@vinc17 yes, the time statistics point to what you said. –  chaos Jul 23 at 12:43
1  
@val0x00ff The < <(tac filename) should be as fast as a pipe: in both cases, the commands run in parallel. –  vinc17 Jul 23 at 12:46
7  
If you're going for efficiency, it would be better to put the tac after the grep. If you've got a 10,000,000 line file, with only 2 matches, tac will only have to reverse 2 lines, not 10m. grep is still going to have to go through the whole thing either way. –  Patrick Jul 23 at 14:10
2  
If you put tac after the grep, it will be reading from a pipe and so can't seek. That will make it less efficient (or fail completely) if the number of found lines is large. –  jjanes Jul 23 at 19:45

This solution might help:

tac file_name | grep -e expression
share|improve this answer
3  
tac is the GNU command. On most other systems, the equivalent is tail -r. –  Stéphane Chazelas Jul 23 at 14:55
    
@Stéphane: On at least some Unix systems, tail -r is limited to a small number of lines, this might be an issue. –  RedGrittyBrick Jul 23 at 16:20
1  
@RedGrittyBrick, do you have any reference for that, or could you please tell which systems have that limitation? –  Stéphane Chazelas Jul 23 at 16:50
    
@StéphaneChazelas, tail -r /etc/passwd fails with tail: invalid option -- 'r'. I'm using coreutils-8.21-21.fc20.x86_64. –  Cristian Ciupitu Jul 23 at 20:14
    
@CristianCiupitu, as I said, GNU has tac (and only GNU has tac) many other Unices have tail -r. GNU tail doesn't support -r –  Stéphane Chazelas Jul 23 at 22:41

If the file is really big, can not fit in memory, I will use Perl with File::ReadBackwards module from CPAN:

$ cat reverse-grep.pl
#!/usr/bin/perl

use strict;                                                                     
use warnings;                                                                   

use File::ReadBackwards;                                                        

my $pattern = shift;                                                                            
my $rev = File::ReadBackwards->new(shift)                                       
    or die "$!";                                                                

while (defined($_ = $rev->readline)) {                                          
    print if /$pattern/;                                                              
}                                                                               

$rev->close;

Then:

$ ./reverse-grep.pl pattern file
share|improve this answer
    
The advantage of this approach is that you can tweak the Perl to do anything you want. –  zzapper Jul 24 at 15:52
1  
@zzapper: It's memory efficient, too, since when it read file line by line instead of slurp file in memory like tac. –  cuonglm Jul 24 at 15:54

This one exits as soon as it finds the first match:

 tac hugeproduction.log | grep -m1 WhatImLookingFor

The following gives the 5 lines before and after the first two matches:

 tac hugeproduction.log | grep -m2 -A 5 -B 5 WhatImLookingFor

Remember not to use -i (case insensitive) unless you have to as that will slow down the grep.

If you know the exact string you are looking for then consider fgrep (Fixed String)

 tac hugeproduction.log | grep -F -m2 -A 5 -B 5 'ABC1234XYZ'
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.