Search for text files where two different words exist (any order, any line)

Question

I'm looking for a way to search files where two word instances exist in the same file. I've been using the following to perform my searches up to this point:

find . -exec grep -l "FIND ME" {} \;

The problem I'm running into is that if there isn't exactly one space that between "FIND" and "ME", the search result does not yield the file. How do I adapt the former search string where both words "FIND" and "ME exist in a file as opposed to "FIND ME"?

I'm using AIX.

Answer 1

With GNU tools:

find . -type f  -exec grep -lZ FIND {} + | xargs -r0 grep -l ME

You can do standardly:

find . -type f -exec grep -q FIND {} \; -exec grep -l ME {} \;

But that would run up to two greps per file. To avoid running that many greps and still be portable while still allowing any character in file names, you could do:

convert_to_xargs() {
  sed "s/[[:blank:]\"\']/\\\\&/g" | awk '
    {
      if (NR > 1) {
        printf "%s", line
        if (!index($0, "//")) printf "\\"
        print ""
      }
      line = $0
    }'
    END { print line }'
}

export LC_ALL=C
find .//. -type f |
  convert_to_xargs |
  xargs grep -l FIND |
  convert_to_xargs |
  xargs grep -l ME

The idea being to convert the output of find into a format suitable for xargs (that expects a blank (SPC/TAB/NL in the C locale, YMMV in other locales) separated list of words where single, double quotes and backslashes can escape blanks and each other).

Generally you can't post-process the output of find -print, because it separates the file names with a newline character and doesn't escape the newline characters that are found in file names. For instance if we see:

./a
./b

We've got no way to know whether it's one file called b in a directory called a<NL>. or if it's the two files a and b in the current directory.

By using .//., because // cannot appear otherwise in a file path as output by find (because there's no such thing as a directory with an empty name and / is not allowed in a file name), we know that if we see a line that contains //, then that's the first line of a new filename. So we can use that awk command to escape all newline characters but those that precede those lines.

If we take the example above, find would output in the first case (one file):

.//a
./b

Which awk escapes to:

.//a\
./b

So that xargs sees it as one argument. And in the second case (two files):

.//a
.//b

Which awk would leave as is, so xargs sees two arguments.

You need the LC_ALL=C so sed, awk (and some implementations of xargs) work for arbitrary sequences of bytes (even though that don't form valid characters in the user's locale), to simplify the blank definition to just SPC and TAB and to avoid problems with different interpretations of characters whose encoding contains the encoding of backslash by the different utilities.

Answer 2

If the files are in a single directory and their name don't contain space, tab, newline, *, ? nor [ characters and don't start with - nor ., this will get a list of files containing ME, then narrow that down to the ones that also contain FIND.

grep -l FIND `grep -l ME *`

Answer 3

With awk you could also run:

find . -type f  -exec awk 'BEGIN{cx=0; cy=0}; /FIND/{cx++}
/ME/{cy++}; END{if (cx > 0 && cy > 0) print FILENAME}' {} \;

It uses cx and cy to count for lines matching FIND and respectively ME. In the END block, if both counters > 0, it prints the FILENAME.
This would be faster/more efficient with gnu awk:

find . -type f  -exec gawk 'BEGINFILE{cx=0; cy=0}; /FIND/{cx++}
/ME/{cy++}; ENDFILE{if (cx > 0 && cy > 0) print FILENAME}' {} +

Answer 4

Or use egrep -e or grep -E like this:

find . -type f -exec egrep -le '(ME.*FIND|FIND.*ME)' {} \;

or

find . -type f -exec grep -lE '(ME.*FIND|FIND.*ME)' {} +

The + makes find (if supported) add multiple file(path)names as arguments to the command being -execed. This saves processes and is a lot quicker than \; which invokes the command once for each file found.

-type f matches only files, to avoid grepping on a directory.

'(ME.*FIND|FIND.*ME)' is a regular expression matching any line containing "ME" followed by "FIND" or "FIND" followed by "ME". (single quotes to prevent the shell from interpreting special characters).

Add a -i to the grep command to make it case-insensitive.

To only match lines where "FIND" comes before "ME", use 'FIND.*ME'.

To require spaces (1 or more, but nothing else) between the words: 'FIND +ME'

To allow spaces (0 or more, but nothing else) between the words: 'FIND *ME'

The combinations are endless with regular expressions, and provided you are interested in matching only on a row-at-a-time basis, egrep is very powerful.

Answer 5

Looking at the accepted answer, it seems more complex than it needs to be. GNU versions of find and grep and xargs support NULL-terminated strings. It's as simple as:

find . -type f -print0 | xargs -0 grep -l --null FIND | xargs -0 grep -l ME

You can modify your find command to filter to the files you wish, and it works with filenames containing any character; without the added complexity of sed parsing. If you wanted to further process the files, add another --null to the last grep

find . -type f -print0 | xargs -0 grep -l --null FIND | xargs -0 grep -l --null ME | xargs -0 echo

And, as a function:

find_strings() {
    find . -type f -print0 | xargs -0 grep -l --null "$1" | xargs -0 grep -l "$2"
}

Obviously, use the accepted answer if you are not running GNU versions of these tools.

Answer 6

TL&DR

Note: You have to test which one is the fastest for yourself.

grep -rlzE '(TermOne.*TermTwo)|(TermTwo.*TermOne)'    # GNU grep

find . -type f -exec grep -q 'TermOne' {} \; \
               -exec grep -q 'TermTwo' {} \; \
               -print

awk '/TermOne/{if(p==0)p=1; if(p==2)p=3}
     /TermTwo/{if(p==0)p=2; if(p==1)p=3}
     p==3{print FILENAME;p=0;nextfile}' ./*

One File

There is no way to build a regex that could match two separate strings in a file.

It is possible to search for two terms with either alternation:

grep -E '(TermOne.*TermTwo)|(TermTwo.*TermOne)' file

or lookahead:

grep -P '(?=.*TermOne)(?=.*TermTwo)' file

but only if the two terms are on the same line

It is also possible to make the whole file act as one file (if the file doesn't contain NULs. Unix text files don't) with the GNU grep -z option:

grep -zE '(TermOne.*TermTwo)|(TermTwo.*TermOne)' file

It is not possible to use -z with -P at the same time, so, no lookahead solutions possible as of today.

The other alternative is to grep twice:

<file grep 'TermOne' | grep -q 'TermTwo'

The exit code of the whole pipe will signal 0 only if both terms were found in one file.

Or, to use awk:

awk '/TermOne/{if(p==0)p=1; if(p==2)p=3}
     /TermTwo/{if(p==0)p=2; if(p==1)p=3}
     p==3{print "both terms found"; exit}' file

list files

The first two solutions from above will work to recursively list all files by adding the options -r (recursive, which then there is no need for a filename), -l (list matching filenames) and -z (assume the whole file is one line).

grep -rlzE '(TermOne.*TermTwo)|(TermTwo.*TermOne)'

Or, using find (two grep calls):

find . -type f -exec grep -q 'TermOne' {} \; \
               -exec grep -q 'TermTwo' {} \; \
               -print

Or, using awk (the glob will include only the PWD):

awk '/TermOne/{if(p==0)p=1; if(p==2)p=3}
     /TermTwo/{if(p==0)p=2; if(p==1)p=3}
     p==3{print FILENAME;p=0;nextfile}' ./*