4

EDIT: I am making this too complicated. It's not about inverse grep. I get the same results using just grep -x -f stop.txt < in.txt. If who comes before whose in the stop word file, the result is just who. When the order is reversed in the stop word file, both lines in in.txt are found. I have the feeling that I fundamentally do not get grep.


I cannot get inverse grep to work like I expect in order to remove lines containing a stop word from a file. The order in which the stop words are given affects the result.

Suppose I have two files. An input file in.txt:

who
whose

And a file with a list of stop words stop.txt:

who
whose

If I "filter" in.txt with an inverse grep search on the stop words in stop.txt, I get:

$ grep -vx -f stop.txt < in.txt
whose
$

Only if I change stop.txt to

whose
who

do I get:

$ grep -vx -f stop.txt < in.txt
$

I do not understand why the order of words in the file with stop words is of importance.

4
  • 3
    Since you are doing whole-line matches, are you sure there is no trailing whitespace or non printing character in either of the files? Check with od or cat -A for example Commented Sep 9, 2022 at 12:40
  • 3
    I can't really replicate this error with the native grep on FreeBSD, nor with GNU grep. Are you sure there are no trailing spaces after whose in any of the files? What Unix are you on and what grep are you using?
    – Kusalananda
    Commented Sep 9, 2022 at 13:02
  • grep (BSD grep) 2.5.1-FreeBSD on macOS 10.15.7. Commented Sep 9, 2022 at 13:11
  • I've thought about trailing whitespace. But does not seem to be the matter. As I said in my comment to Marcus's answer, replacing who and whose with a and b leads to the expected result. It looks like the behavior has to do with the fact that who is a substring of whose, no? Commented Sep 9, 2022 at 13:14

1 Answer 1

5

With the pattern files

$ od -bc shortlong
0000000   167 150 157 012 167 150 157 163 145 012
           w   h   o  \n   w   h   o   s   e  \n
0000012
$ od -bc longshort
0000000   167 150 157 163 145 012 167 150 157 012
           w   h   o   s   e  \n   w   h   o  \n
0000012

we can test some of the variations, here with the macOS grep (2.5.1-FreeBSD):

$ grep -x -f shortlong shortlong
who
$ grep -x -f shortlong longshort
who
$ grep -x -f longshort shortlong
who
whose
$ grep -x -f longshort longshort
whose
who

who when first hides the longer whose match. This appears to be a bug.

The problem only appears when the expressions come from a -f file, not when the equivalent (we hope) expression is given as an argument:

$ grep -x -E 'who|whose' shortlong
who
whose
$ grep -x -E 'who|whose' longshort
whose
who
$ grep -x -E 'whose|who' shortlong
who
whose
$ grep -x -E 'whose|who' longshort
whose
who

GNU grep (3.7) does not exhibit this problem (nor does the grep on current versions of OpenBSD):

$ ggrep -x -f shortlong shortlong
who
whose
$ ggrep -x -f shortlong longshort
whose
who
$ ggrep -x -f longshort shortlong
who
whose
$ ggrep -x -f longshort longshort
whose
who

Therefore to avoid this bug do not use grep verion 2.5.1-FreeBSD, or instead form the regular expressions into an argument from the file by replacing non-ultimate \n with | for grep -E.

$ paste -s -d \| shortlong
who|whose
$ grep -x -E -- "$(paste -s -d \| shortlong)" shortlong
who
whose

Concerning Alternation

Longer strings should ideally be listed before shorter strings in regular expression alternations, among other edge cases, which means one should prefer the longshort file or whose|who forms.

grep appears (when it is not buggy) to get this right, but other regular expression engines will stop on the first match, which must be listed longest first if you want those to match:

$ printf 'who whoses the whosefolk' |
grep -o -E 'who|whose|whosefolk'
who
whose
whosefolk
$ printf 'who whoses the whosefolk' |
perl -nle 'print for /who|whose|whosefolk/g'
who
who
who
$ printf 'who whoses the whosefolk' |
perl -nle 'print for /whosefolk|whose|who/g'
who
whose
whosefolk
3
  • Thank you very much! I suspected it had to do with the fact that who is a substring of whose but I very much lack the technical expertise to confirm this as a possible bug. Commented Sep 9, 2022 at 17:12
  • strictly speaking, \| is not part of basic regexes, even if some support it. The re_format man page on my mac describes ERE first, and then says "Obsolete (''basic'') regular expressions differ in several respects. '|' is an ordinary character and there is no equivalent for its functionality." but it also admits an extensions: "When the REG_ENHANCED flag is passed [...], additional features are activated. [...] For enhanced basic REs, '+', '?' and '|' remain regular characters, but '\+', '\?' and `\|' have the same special meaning as the unescaped characters do for extended REs"
    – ilkkachu
    Commented Sep 9, 2022 at 19:46
  • I think reverse sorting (sort -r) the file with the patterns might work to ensure that no pattern is a prefix of a latter one if that's indeed enough to prevent the issue.
    – ilkkachu
    Commented Sep 9, 2022 at 19:49

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .