0

Please advise why grep -f works faster than grepping multiple keywords indidiuvally on same file.

Some cases, both results are produced in seconds. On certain files, normal grep takes 15 mins and grep -f takes less than a minute.

grep -f <search_pattern_50keys_file.txt> <file_name>
grep search_key_1 <file_name>
grep search_key_1 <file_name>
...
...
grep search_key_50 <file_name>

Regards, Veera

Improve this question
3
  • 6
    At the very least 50 greps reading the same line once each is going to be slower than 1 grep reading that line once.
    – muru
    Commented May 10, 2021 at 16:16
  • 1
    might be mildly interesting to see how the elapsed times change as a function of the file size...
    – ilkkachu
    Commented May 10, 2021 at 19:24
  • A small file will be completely cached, so second and subsequent reads will be RAM-accessed. As soon as the file size gets close to the available memory for caching, the cache will be cyclically replaced and the data will be read from HDD every time. That's going to be a hundred times slower.
    – Paul_Pedant
    Commented May 10, 2021 at 21:08

1 Answer 1

Reset to default
0

There is no real difference in execution time in my experience. I tried with a very large text file. My pattern file contained:

2a8f

and

$ time grep 2a8f /tmp/md5sums > /dev/null

real    0m0,543s
user    0m0,439s
sys     0m0,104s

versus

$ time grep -f pattern  /tmp/md5sums > /dev/null

real    0m0,544s
user    0m0,438s
sys     0m0,106s

Note that the half second is per grep. So, of course, if you would do 50 greps with different patterns, that would be 25 seconds, whereas a single grep with 50 patterns would extend the time much less since it would have to go over the file (/tmp/md5sums in my case) just once.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .