Let's say, I have a really big text file (about 10.000.000 lines). I need to grep it from the end and save result to a file. What's the most efficient way to accomplish task?

  • 10
    Use tac and grep to achieve what you want. – Valentin Bajrami Jul 23 '14 at 12:06
  • 1
    In addition to the excellent solutions posted, GNU grep has a --max-count (number) switch that aborts after a certain number of matches, which might be interesting to you. – Ulrich Schwarz Jul 23 '14 at 13:28
  • @val0x00ff could you take a look at this question – c0rp Jul 24 '14 at 8:50
  • Do you know how much hits you will have? When you think your grep will find 3 lines, start grepping and reverse afterwards. – Walter A May 1 '15 at 19:24

tac/grep Solution

tac file | grep whatever

Or a bit more effective:

grep whatever < <(tac file)

Time with a 500MB file:

real    0m1.225s
user    0m1.164s
sys     0m0.516s

sed/grep Solution:

sed '1!G;h;$!d' | grep whatever

Time with a 500MB file: Aborted after 10+ minutes.

awk/grep Solution:

awk '{x[NR]=$0}END{while (NR) print x[NR--]}' file | grep whatever

Time with a 500MB file:

real    0m5.626s
user    0m4.964s
sys     0m1.420s

perl/grep Solution:

perl -e 'print reverse <>' file | grep whatever

Time with a 500MB file:

real    0m3.551s
user    0m3.104s
sys     0m1.036s
  • 2
    sed, awk and perl (with this method) are not OK since they read the file from the beginning, which is very inefficient. I suppose that tac does the right thing. – vinc17 Jul 23 '14 at 12:39
  • 1
    @vinc17 yes, the time statistics point to what you said. – chaos Jul 23 '14 at 12:43
  • 1
    @val0x00ff The < <(tac filename) should be as fast as a pipe: in both cases, the commands run in parallel. – vinc17 Jul 23 '14 at 12:46
  • 7
    If you're going for efficiency, it would be better to put the tac after the grep. If you've got a 10,000,000 line file, with only 2 matches, tac will only have to reverse 2 lines, not 10m. grep is still going to have to go through the whole thing either way. – Patrick Jul 23 '14 at 14:10
  • 3
    If you put tac after the grep, it will be reading from a pipe and so can't seek. That will make it less efficient (or fail completely) if the number of found lines is large. – jjanes Jul 23 '14 at 19:45

This solution might help:

tac file_name | grep -e expression
  • 3
    tac is the GNU command. On most other systems, the equivalent is tail -r. – Stéphane Chazelas Jul 23 '14 at 14:55
  • @Stéphane: On at least some Unix systems, tail -r is limited to a small number of lines, this might be an issue. – RedGrittyBrick Jul 23 '14 at 16:20
  • 1
    @RedGrittyBrick, do you have any reference for that, or could you please tell which systems have that limitation? – Stéphane Chazelas Jul 23 '14 at 16:50
  • @StéphaneChazelas, tail -r /etc/passwd fails with tail: invalid option -- 'r'. I'm using coreutils-8.21-21.fc20.x86_64. – Cristian Ciupitu Jul 23 '14 at 20:14
  • @CristianCiupitu, as I said, GNU has tac (and only GNU has tac) many other Unices have tail -r. GNU tail doesn't support -r – Stéphane Chazelas Jul 23 '14 at 22:41

This one exits as soon as it finds the first match:

 tac hugeproduction.log | grep -m1 WhatImLookingFor

The following gives the 5 lines before and after the first two matches:

 tac hugeproduction.log | grep -m2 -A 5 -B 5 WhatImLookingFor

Remember not to use -i (case insensitive) unless you have to as that will slow down the grep.

If you know the exact string you are looking for then consider fgrep (Fixed String)

 tac hugeproduction.log | grep -F -m2 -A 5 -B 5 'ABC1234XYZ'

If the file is really big, can not fit in memory, I will use Perl with File::ReadBackwards module from CPAN:

$ cat reverse-grep.pl
#!/usr/bin/perl

use strict;
use warnings;

use File::ReadBackwards;

my $pattern = shift;
my $rev = File::ReadBackwards->new(shift)
    or die "$!";

while (defined($_ = $rev->readline)) {
    print if /$pattern/;
}

$rev->close;

Then:

$ ./reverse-grep.pl pattern file
  • The advantage of this approach is that you can tweak the Perl to do anything you want. – zzapper Jul 24 '14 at 15:52
  • 1
    @zzapper: It's memory efficient, too, since when it read file line by line instead of slurp file in memory like tac. – cuonglm Jul 24 '14 at 15:54
  • can anyone add a -m support for this ? I'd like to test in on real files. See : gist.githubusercontent.com/ychaouche/… – ychaouche Nov 5 at 14:29

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.