44

Let's say, I have a really big text file (about 10.000.000 lines). I need to grep it from the end and save result to a file. What's the most efficient way to accomplish task?

  • 10
    Use tac and grep to achieve what you want. – Valentin Bajrami Jul 23 '14 at 12:06
  • 1
    In addition to the excellent solutions posted, GNU grep has a --max-count (number) switch that aborts after a certain number of matches, which might be interesting to you. – Ulrich Schwarz Jul 23 '14 at 13:28
  • @val0x00ff could you take a look at this question – c0rp Jul 24 '14 at 8:50
  • Do you know how much hits you will have? When you think your grep will find 3 lines, start grepping and reverse afterwards. – Walter A May 1 '15 at 19:24
46

tac/grep Solution

tac file | grep whatever

Or a bit more effective:

grep whatever < <(tac file)

Time with a 500MB file:

real    0m1.225s
user    0m1.164s
sys     0m0.516s

sed/grep Solution:

sed '1!G;h;$!d' | grep whatever

Time with a 500MB file: Aborted after 10+ minutes.

awk/grep Solution:

awk '{x[NR]=$0}END{while (NR) print x[NR--]}' file | grep whatever

Time with a 500MB file:

real    0m5.626s
user    0m4.964s
sys     0m1.420s

perl/grep Solution:

perl -e 'print reverse <>' file | grep whatever

Time with a 500MB file:

real    0m3.551s
user    0m3.104s
sys     0m1.036s
  • 2
    sed, awk and perl (with this method) are not OK since they read the file from the beginning, which is very inefficient. I suppose that tac does the right thing. – vinc17 Jul 23 '14 at 12:39
  • 1
    @vinc17 yes, the time statistics point to what you said. – chaos Jul 23 '14 at 12:43
  • 1
    @val0x00ff The < <(tac filename) should be as fast as a pipe: in both cases, the commands run in parallel. – vinc17 Jul 23 '14 at 12:46
  • 7
    If you're going for efficiency, it would be better to put the tac after the grep. If you've got a 10,000,000 line file, with only 2 matches, tac will only have to reverse 2 lines, not 10m. grep is still going to have to go through the whole thing either way. – Patrick Jul 23 '14 at 14:10
  • 3
    If you put tac after the grep, it will be reading from a pipe and so can't seek. That will make it less efficient (or fail completely) if the number of found lines is large. – jjanes Jul 23 '14 at 19:45
17

This solution might help:

tac file_name | grep -e expression
  • 3
    tac is the GNU command. On most other systems, the equivalent is tail -r. – Stéphane Chazelas Jul 23 '14 at 14:55
  • @Stéphane: On at least some Unix systems, tail -r is limited to a small number of lines, this might be an issue. – RedGrittyBrick Jul 23 '14 at 16:20
  • 1
    @RedGrittyBrick, do you have any reference for that, or could you please tell which systems have that limitation? – Stéphane Chazelas Jul 23 '14 at 16:50
  • @StéphaneChazelas, tail -r /etc/passwd fails with tail: invalid option -- 'r'. I'm using coreutils-8.21-21.fc20.x86_64. – Cristian Ciupitu Jul 23 '14 at 20:14
  • @CristianCiupitu, as I said, GNU has tac (and only GNU has tac) many other Unices have tail -r. GNU tail doesn't support -r – Stéphane Chazelas Jul 23 '14 at 22:41
10

This one exits as soon as it finds the first match:

 tac hugeproduction.log | grep -m1 WhatImLookingFor

The following gives the 5 lines before and after the first two matches:

 tac hugeproduction.log | grep -m2 -A 5 -B 5 WhatImLookingFor

Remember not to use -i (case insensitive) unless you have to as that will slow down the grep.

If you know the exact string you are looking for then consider fgrep (Fixed String)

 tac hugeproduction.log | grep -F -m2 -A 5 -B 5 'ABC1234XYZ'
9

If the file is really big, can not fit in memory, I will use Perl with File::ReadBackwards module from CPAN:

$ cat reverse-grep.pl
#!/usr/bin/perl

use strict;
use warnings;

use File::ReadBackwards;

my $pattern = shift;
my $rev = File::ReadBackwards->new(shift)
    or die "$!";

while (defined($_ = $rev->readline)) {
    print if /$pattern/;
}

$rev->close;

Then:

$ ./reverse-grep.pl pattern file
  • The advantage of this approach is that you can tweak the Perl to do anything you want. – zzapper Jul 24 '14 at 15:52
  • 1
    @zzapper: It's memory efficient, too, since when it read file line by line instead of slurp file in memory like tac. – cuonglm Jul 24 '14 at 15:54
  • can anyone add a -m support for this ? I'd like to test in on real files. See : gist.githubusercontent.com/ychaouche/… – ychaouche Nov 5 '18 at 14:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy