2

I'm trying to search through my email backup for an important email. It's a directory with subdirectories that contains a few thousand .eml file (on a Linux filesystem). I would like to search for .eml text files that contain three words and to exclude one word.

First I tried searching for one word, then another with piping.

grep -R 'foo' ~/Directory/path | grep 'bar'

That didn't work, as it only returned files that contained the two words on the same line. I needed files that contained two words in the whole file.

I tried finding files that contained one word and piped the file contents to an output file.

grep -rIlZ  '.' -e 'foo' | xargs -0 cat > MyOutputFile 

That was helpful, as I could see the context. But I needed to search for more than one word. Is it possbible to expand this to search for more than one word, and to exclude one word?

2

4 Answers 4

5

Suppose we want names of files that contain foo and bar but not baz. In that case:

find . -type f -exec gawk '
  BEGINFILE{a=b=c=0}
  /foo/{a=1} /bar/{b=1} /baz/{c=1;nextfile}
  ENDFILE{if(a && b && !c)print FILENAME}' {} +

[Since you are on Linux, I assume you have ready access to GNU awk (gawk).]

Note that, in this approach, as few awk invocations as possible are started and each file is read only once. No intermediate files are needed. This should be efficient.

Example

Let's consider a directory with two files:

$ cat file1.eml 
foo and
bar only
$ cat file2.eml 
foo
and
bar
and
baz

If we run our command, it produces ./file1.eml as the only file that meets the requirements:

$ find . -type f -exec gawk '
    BEGINFILE{a=b=c=0}
    /foo/{a=1} /bar/{b=1} /baz/{c=1;nextfile}
    ENDFILE{if(a && b && !c)print FILENAME}' {} +
./file1.eml

How it works

  • find collects the list of regular files recursively and passes it gawk.

  • BEGINFILE{a=b=c=0}

    At the start of every new file, this sets variables a, b, and c to zero (false).

  • /foo/{a=1}

    If any line contains foo, set variable a to one. (true).

  • /bar/{b=1}

    If any line contains bar, set variable b to one. (true).

  • /baz/{c=1;nextfile}

    If any line contains baz, set variable c to one. (true).

    After any word to exclude if found, such as baz in our example, there is no point reading any more of the file. So, we run nextfile to skip the rest of the lines and go immediately to ENDFILE.

  • ENDFILE{if(a && b && !c)print FILENAME}

    At the end of each file, if a and b and not c (in awk ! is logical-not) are all true, then print the file's name.

Non-GNU awk

If your awk doesn't have the nice BEGINFILE and ENDFILE features, like mawk, you'd need to run one awk per file:

find . -type f -exec mawk '
  /foo/{a=1} /bar/{b=1} /baz/{c=1;exit}
  END{if(a && b && !c) print FILENAME}' {} \;

or (hat tip: Ed Morton):

awk 'FNR==1 { if (a && b && !c) print fname; fname=FILENAME; a=b=c=0 } /foo/{a=1} /bar/{b=1} /baz/{c=1}   END{if(a && b && !c) print FILENAME}' *.eml

or, with recursive search:

find . -type f -exec awk 'FNR==1 { if (a && b && !c) print fname; fname=FILENAME; a=b=c=0 } /foo/{a=1} /bar/{b=1} /baz/{c=1}   END{if(a && b && !c) print FILENAME}' {} +
8
  • @StéphaneChazelas Both excellent suggestions! Answer updated with the ./ precaution and the nextfile trick.
    – John1024
    Jul 27, 2020 at 7:32
  • 1
    @AaronMcMillin {} + puts many file names on each command line. {} \; puts only one on each command line (requiring many more invocations of the command with all the associated overhead of starting and ending processes).
    – John1024
    Jul 27, 2020 at 19:56
  • 1
    See also grep'ping files for multiple strings (not necessarily on the same line) for a generalisation (though without the negative matches) of that approach for a given list of patterns. Jul 27, 2020 at 20:23
  • 1
    @EdMorton Answer updated with your FNR==1... code.
    – John1024
    Jul 28, 2020 at 21:12
  • 1
    @EdMorton Ugh! Yes! Corrected (I think).
    – John1024
    Jul 28, 2020 at 21:38
2

Try find -exec with grep -q:

find /my/path -name "*.eml" \
  -exec grep -F -q "word1" {} \; \
  -exec grep -F -q "word2" {} \; \
  -exec grep -F -q "word3" {} \; \
  ! -exec grep -F -q "word4" {} \; \
  -print
  • grep -q returns only status code
  • Leave out the -F from grep if you want to search patterns instead of words
  • Add -w to grep to match only whole words: match word but not someword.
  • find chains the -exec commands and stop when one of it fails (when grep -q returns an error code)
0
2

You can use an approach like:

grep -rIlZe foo . |
  xargs -r0 grep -lZe bar |
  xargs -r0 grep -LZe baz |
  xargs -r0 cat > MyOutputFile

That is, feed the list of files generated by the first grep to xargs -r0 to pass to the next grep which further refines the list.

Note the -L option for the last grep which is like -l except that it reports the files where no match is found, so we end up with the files that contain foo and bar and not baz.

The -r and -I are only needed or the first grep. The latter ones will get lists of regular files as arguments (with binary files already filtered out by -I on the first grep), not directories for which to recurse in.

That means the contents of files could end up being read several times, which is not very efficient, but grep implementations being generally a lot faster than awk implementations, and also with the fact that since all 4 commands above are started in parallel, some of that processing will be performed concurrently by several processors and with data already cached in memory, it's likely to be faster than awk-based ones.

1
  • Nicely done combination of grep and xargs.
    – John1024
    Jul 27, 2020 at 19:59
0

Just copy and paste this piece of code into a new bash script file, save it and do chmod +x <file> then run it in your terminal to list all files that contain "foo" and "bar" and don't contain "rab" strings:

#!/bin/bash
function notcontain {
        for FILE in $(find . 2> /dev/null); do
                if ! grep "rab" $FILE > /dev/null 2>&1; then
                        echo $FILE
                fi
        done
}
    
for FILE in `notcontain`; do
        if grep "foo" $FILE > /dev/null 2>&1 | grep "bar" $FILE > /dev/null 2>&1; then
                echo $FILE
        fi
done

Hope this will help :)

2
  • 5
    Welcome to the site, and thank you for your contribution. Please note, however, that this would locate files that contain either foo or bar whereas the OP wants to look for files that contain both foo and bar (and a third word, actually ;) ...)
    – AdminBee
    Jul 27, 2020 at 9:20
  • @AdminBee Thanks for the tips. I edited the answer.
    – Farhad Kia
    Jul 29, 2020 at 13:27

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .