17

So I have a bunch of Apache logs with using the standard log format. I want to get all the log lines that did not come from a web crawler.

So lets say I have a file robot_patterns with entries like

Googlebot
msnbot-media
YandexBot
bingbot

If I run the command grep -f robot_patterns *.log I will get all the entries by bots matching the above patterns. My actual list has ~30 entries of bots and agents that I wish to ignore.

But I want to find all the entries that are NOT from bots. So I try grep -v -f robot_patterns *.log and no results are returned by grep. This is not what I expect or desire, and I am not finding an obvious way to get what I want. When using the -v option combined with multiple patterns in a file, grep will only return a matching line if it matches EVERY pattern.

4
  • When I tried this on my system, grep -v -f had the desired behavior, only returning lines that matched none of the patterns. This was with (GNU grep) 2.14.56-1e3d. What grep are you using?
    – wingedsubmariner
    Dec 11, 2013 at 23:36
  • I am running GNU grep 2.6.3.
    – Zoredache
    Dec 11, 2013 at 23:42
  • 4
    I did some more testing, and found that if there is an empty line in the patterns file it will match every line, causing no lines to be returned with -v. This isn't a problem with -F however, and -F might speed up grep for your task - might be worth trying for you.
    – wingedsubmariner
    Dec 11, 2013 at 23:59
  • A trailing empty line! Argh... That seems to be be the issue. If you want, you should add that as answer.
    – Zoredache
    Dec 12, 2013 at 0:10

3 Answers 3

Reset to default
20

You can try:

grep -vE 'Googlebot|msnbot-media|YandexBot|bingbot' yourlogfile
4
  • 2
    Welcome to Unix & Linux.  The OP has a list of approximately 30 strings that he wants to ignore, and the four that he presented as examples have an average length of ten characters each, so your command is likely to be over 300 characters long.  This is likely to be hard to maintain (and even to read).  Can you modify your answer to be driven by the OP’s list of strings?  … … … … … … … … … … … … … … … … … … … …  P.S. Did you notice that the answer has been found? — The OP has learned how to get his original approach to work.
    – G-Man Says 'Reinstate Monica'
    Oct 30, 2015 at 11:33
  • 2
    Why negatively evaluate my response? : /
    – Orsius
    Oct 31, 2015 at 10:17
  • 3
    Great answer. Has regex OR and the -vE option was helpful.
    – Kirt Carson
    Mar 31, 2017 at 18:46
  • 4
    This is the answer to the question most people are probably trying to resolve.
    – Dominik Stańczak
    Jun 15, 2018 at 7:57
12

If there is an empty line in the patterns file it will match every line, causing no lines to be returned with -v. This is because the lines are interpreted as regular expressions, and an empty regular expression will always match.

This isn't a problem with -F however, because grep ignores empty lines with -F.
-F causes grep to interpret the lines as simple strings to search for and may speed up grep if regular expressions aren't needed.

1
0

The way I do it is to chain the greps:

cat text | grep -v Googlebot | grep -v msnbot-media | grep -v YandexBot | grep -v bingbot

1
  • Until now I did the same, but I believe, that grep -v -e Googlebot -e YandexBot ... is better. There is just one grep and 1 stream... You do not need first cat at all, user grep -v Googlebot text (as text is a file in this case IMHO)
    – Betlista
    Nov 16 at 10:49

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .