How to search for multiline text files containing a set of words (e.g., AAA & (BBB | CCC) & ~DD)?

Question

I need to find the files that fulfill relatively complex condition. For example, I want to find all files that fulfill all below conditions:

• does contain word AAAA
• does contain word BBB or CCCCC (may contain both of them)
• does not contain word DDD

The words may appear in any order and may appear in different lines.

I have one solution, that combines find and egrep, but is not very legible.

find . \( -type f -and -exec egrep -q 'BBB|CCCCC' {} \; \
   -and -exec egrep -q AAAA {} \; \
   -and -not -exec egrep -q DDD {} \; \) -print

Is there any better way to solve that problem?

Answer 1

Your solution is pretty legible for the task, in my opinion. However, it's slow, because it spawns 3 processes per file. I reckon Awk is better suited here because it will allow to read a whole batch of files (as allowed by ARG_MAX) in a single go, using {} + instead of {} \;.

GNU Awk:

find . -type f -exec gawk '
    BEGINFILE{c1=c2=c3=0}
    /AAA/       {c1=1}
    /BBB/||/CCC/{c2=1}
    /DDD/       {c3=1; nextfile}
    ENDFILE{if(c1 && c2 && !c3)print FILENAME}
' {} +

POSIX^*:

find . -type f -exec awk '
    FNR==1{
        if(NR>1 && c1 && c2 && !c3)print f
        c1=c2=c3=0
        f=FILENAME
    }
    /AAA/       {c1=1}
    /BBB/||/CCC/{c2=1}
    /DDD/       {c3=1; nextfile}
    END{if(c1 && c2 && !c3)print f}
' {} +

^{*Actually, nextfile is still not POSIX but it has been accepted to the next issue of the standard. You can remove it for POSIX issue 7 compliance; the result will be the same, but with a performance penalty.}

---

Note: Awk bails out if it does not have permissions to read a file. In GNU Find, simply add the -readable flag to avoid that. If GNU Find is not available, Test can be used as an additional filter:

find . -type f -exec test -r {} \; -exec awk '
    ...
' {} +

But spawning a Test for each file represents a performance penalty.

---

Further reading:

• POSIX Find.
• POSIX Awk.
• The BEGINFILE/ENDFILE special patterns.

Answer 2

Here is an alternative way. It works by first making lists of files that contain each individual word:

find . -type f -exec grep -lF 'AAA' {} + > files_with_AAA
find . -type f -exec grep -lF 'BBB' {} + > files_with_BBB
...

Then the conditions can be implemented by processing the lists:

grep -Ff files_with_BBB files_with_AAA   # AAA & BBB
grep -Fvf files_with_BBB files_with_AAA  # AAA & ~BBB
sort -u files_with_AAA files_with_BBB    # AAA | BBB

If there is a large amount of files and multiple expressions to evaluate, this can be faster as each file does not have to be rescanned.

Answer 3

I'd be interested to see a timing comparison with these solutions, especially for ripgrep which has built-in parallel processing.

1. With GNU grep

grep -rLZ 'DDD' | xargs -0 grep -lZ 'AAAA' | xargs -0 grep -lE 'BBB|CCCCC'

# if your search terms are literal strings
grep -rLZF 'DDD' | xargs -0 grep -lZF 'AAAA' | xargs -0 grep -lF -e 'BBB' -e 'CCCCC'

2. With rg . Note that recursive search is enabled by default and certain files are also ignored by default. You need to use -u if you don't want files like .gitignore to influence results. Use -uu to additionally search hidden files. Use -uuu if you want to search binary files as well.

rg --files-without-match -0 'DDD' | xargs -0 rg -l0 'AAAA' | xargs -0 rg -l 'BBB|CCCCC'

# if your search terms are literal strings
rg --files-without-match -0F 'DDD' | xargs -0 rg -l0F 'AAAA' | xargs -0 rg -lF -e 'BBB' -e 'CCCCC'

3. With rg multiline matching

rg -lUP '(?s)\A(?!.*DDD)(?=.*(BBB|CCCCC)).*AAAA'