9

I need to find the files that fulfill relatively complex condition. For example, I want to find all files that fulfill all below conditions:

  • does contain word AAAA
  • does contain word BBB or CCCCC (may contain both of them)
  • does not contain word DDD

The words may appear in any order and may appear in different lines.

I have one solution, that combines find and egrep, but is not very legible.

find . \( -type f -and -exec egrep -q 'BBB|CCCCC' {} \; \
   -and -exec egrep -q AAAA {} \; \
   -and -not -exec egrep -q DDD {} \; \) -print

Is there any better way to solve that problem?

3
  • 2
    How big are the files? In particular are they each small enough to sit in memory? In general you want to only read the files once, rather than 3 times. This suggests to me to use awk/perl/python. If I stick with your approach I would use additional flags to egrep if available. In particular -q to stop as soon as the match has been determined. I would use -v to egrep rather than -not in find as I consider it more readable.
    – icarus
    Jun 25 at 15:51
  • Usually I need that functionality to search for certain source files (C, Python, C++) the use certain specific libraries or solutions. So the file should fit in the memory, however avoiding a necessity to read the whole file may significantly speed up the search (that's why I used -q option).
    – wzab
    Jun 25 at 16:09
  • 1
    as an aside, that looks like all the actions are chained with and, so you shouldn't need the parentheses, or indeed the -and, since it's implicit between any actions if you don't specify -or. (Like just before the -print.) And as a nitpick, -and and -not aren't POSIX, the standard ones are -a, -o and !. So, just find . -type f -exec grep ... \; -exec grep ... \; ! -exec grep ... \; -print might do.
    – ilkkachu
    Jun 26 at 8:41
15

Your solution is pretty legible for the task, in my opinion. However, it's slow, because it spawns 3 processes per file. I reckon Awk is better suited here because it will allow to read a whole batch of files (as allowed by ARG_MAX) in a single go, using {} + instead of {} \;.

GNU Awk:

find . -type f -exec gawk '
    BEGINFILE{c1=c2=c3=0}
    /AAA/       {c1=1}
    /BBB/||/CCC/{c2=1}
    /DDD/       {c3=1; nextfile}
    ENDFILE{if(c1 && c2 && !c3)print FILENAME}
' {} +

POSIX*:

find . -type f -exec awk '
    FNR==1{
        if(NR>1 && c1 && c2 && !c3)print f
        c1=c2=c3=0
        f=FILENAME
    }
    /AAA/       {c1=1}
    /BBB/||/CCC/{c2=1}
    /DDD/       {c3=1; nextfile}
    END{if(c1 && c2 && !c3)print f}
' {} +

*Actually, nextfile is still not POSIX but it has been accepted to the next issue of the standard. You can remove it for POSIX issue 7 compliance; the result will be the same, but with a performance penalty.


Note: Awk bails out if it does not have permissions to read a file. In GNU Find, simply add the -readable flag to avoid that. If GNU Find is not available, Test can be used as an additional filter:

find . -type f -exec test -r {} \; -exec awk '
    ...
' {} +

But spawning a Test for each file represents a performance penalty.


Further reading:

1
  • 1
    @EdMorton That f && was meant to keep the outcome correct even if someone uncritically modified the logic so as to test for only negative conditions (!c1 && !c2 && !c3). Ironically, it actually introduced a bug (the one you point out), so I've adopted your superior suggestion, NR>1. Many thanks for keeping an watchful eye on Awk answers!
    – Quasímodo
    Jul 5 at 12:41
2

Here is an alternative way. It works by first making lists of files that contain each individual word:

find . -type f -exec grep -lF 'AAA' {} + > files_with_AAA
find . -type f -exec grep -lF 'BBB' {} + > files_with_BBB
...

Then the conditions can be implemented by processing the lists:

grep -Ff files_with_BBB files_with_AAA   # AAA & BBB
grep -Fvf files_with_BBB files_with_AAA  # AAA & ~BBB
sort -u files_with_AAA files_with_BBB    # AAA | BBB

If there is a large amount of files and multiple expressions to evaluate, this can be faster as each file does not have to be rescanned.

1

I'd be interested to see a timing comparison with these solutions, especially for ripgrep which has built-in parallel processing.

  1. With GNU grep
grep -rLZ 'DDD' | xargs -0 grep -lZ 'AAAA' | xargs -0 grep -lE 'BBB|CCCCC'

# if your search terms are literal strings
grep -rLZF 'DDD' | xargs -0 grep -lZF 'AAAA' | xargs -0 grep -lF -e 'BBB' -e 'CCCCC'
  1. With rg . Note that recursive search is enabled by default and certain files are also ignored by default. You need to use -u if you don't want files like .gitignore to influence results. Use -uu to additionally search hidden files. Use -uuu if you want to search binary files as well.
rg --files-without-match -0 'DDD' | xargs -0 rg -l0 'AAAA' | xargs -0 rg -l 'BBB|CCCCC'

# if your search terms are literal strings
rg --files-without-match -0F 'DDD' | xargs -0 rg -l0F 'AAAA' | xargs -0 rg -lF -e 'BBB' -e 'CCCCC'
  1. With rg multiline matching
rg -lUP '(?s)\A(?!.*DDD)(?=.*(BBB|CCCCC)).*AAAA'

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.