172

I would like to get the multi pattern match with implicit AND between patterns, i.e. equivalent to running several greps in a sequence:

grep pattern1 | grep pattern2 | ...

So how to convert it to something like?

grep pattern1 & pattern2 & pattern3

I would like to use single grep because I am building arguments dynamically, so everything has to fit in one string. Using filter is system feature, not grep, so it is not an argument for it.


Don't confuse this question with:

grep "pattern1\|pattern2\|..."

This is an OR multi pattern match. I am looking for an AND pattern match.

3

11 Answers 11

146

To find the lines that match each and everyone of a list of patterns, agrep (the original one, now shipped with glimpse, not the unrelated one in the TRE regexp library) can do it with this syntax:

agrep 'pattern1;pattern2'

With GNU grep, when built with PCRE support, you can do:

grep -P '^(?=.*pattern1)(?=.*pattern2)'

With ast grep:

grep -X '.*pattern1.*&.*pattern2.*'

(adding .*s as <x>&<y> matches strings that match both <x> and <y> exactly, a&b would never match as there's no such string that can be both a and b at the same time).

If the patterns don't overlap, you may also be able to do:

grep -e 'pattern1.*pattern2' -e 'pattern2.*pattern1'

The best portable way is probably with awk as already mentioned:

awk '/pattern1/ && /pattern2/'

Or with sed:

sed -e '/pattern1/!d' -e '/pattern2/!d'

Or perl:

perl -ne 'print if /pattern1/ && /pattern2/'

Please beware that all those will have different regular expression syntaxes.

The awk/sed/perl ones don't reflect whether any line matched the patterns in their exit status. To so that you need:

awk '/pattern1/ && /pattern2/ {print; found = 1}
     END {exit !found}'
perl -ne 'if (/pattern1/ && /pattern2/) {print; $found = 1}
          END {exit !$found}'

Or pipe the command to grep '^'.

For potentially gzip-compressed files, you can use zgrep which is generally a shell script wrapper around grep, and use one of the grep solutions above (not the ast-open one as that grep implementation cannot be use by zgrep) or you could use the PerlIO::gzip module of perl which can transparently uncompress files upon input:

perl -MPerlIO::gzip -Mopen='IN,gzip(autopop)' -ne '
  print "$ARGV:$_" if /pattern1/ && /pattern2/' -- *.gz

(which if the files are small enough at least is even going to be more efficient than zgrep as the decompression is done internally without having to run gunzip for each file).

18
  • 4
    The agrep syntax is not working for me... which version was it introduced in?
    – Raman
    Commented Sep 5, 2016 at 22:15
  • 2
    @Raman, yours sounds like TRE agrep. Commented Sep 6, 2016 at 7:01
  • 3
    @Techiee, or just awk '/p1/ && /p2/ {n++}; END {print 0+n}' Commented Jun 28, 2017 at 20:23
  • 1
    @ChamindaBandara, you ran that with GNU grep instead of ast grep. GNU grep has no support for ast augmented regexp. It does have an undocumented -X option, but that's for something unrelated, it's to specify the regexp flavour (matcher) like in grep -X perl being the same as grep -P. Commented Feb 17, 2021 at 9:00
  • 2
    @DanielKaplan, from your recent question, I suspect you're looking for something difference from what this Q&A is about. Here we're trying to find lines that match all patterns, while you may be trying to find files for which all patterns are matched by any line (there are several Q&As here covering that). I've edited the answer to maybe make that more obvious. Commented Mar 10, 2022 at 7:32
30

You didn't specify grep version, this is important. Some regexp engines allow multiple matching groupped by AND using '&' but this is non-standard and non-portable feature. But, at least GNU grep doesn't support this.

OTOH you can simply replace grep with sed, awk, perl, etc. (listed in order of weight increasing). With awk, the command would look like

awk '/regexp1/ && /regexp2/ && /regexp3/ { print; }'

and it can be constructed to be specified in command line in easy way.

7
  • 5
    Just remember that awk uses ERE's, e.g. the equivalent of grep -E, as opposed to the BRE's that plain grep uses.
    – jw013
    Commented Nov 10, 2012 at 9:42
  • 4
    awk's regexes are called EREs, but in fact they're a bit idiosyncratic. Here are probably more details than anyone cares for: wiki.alpinelinux.org/wiki/Regex
    – dubiousjim
    Commented Nov 10, 2012 at 15:35
  • Thank you, grep 2.7.3 (openSUSE). I upvoted you, but I will keep question open for a while, maybe there is some trick for grep (not that I dislike awk -- simply knowing more is better). Commented Nov 10, 2012 at 15:42
  • 3
    The default action is to print the matching line so the { print; } part isn't really necessary or useful here.
    – tripleee
    Commented Apr 20, 2017 at 11:58
  • This still returns a 0 status code if the match fails. Commented Jul 14, 2023 at 4:48
19

git grep

Here is the syntax using git grep combining multiple patterns using Boolean expressions:

git grep --no-index -e pattern1 --and -e pattern2 --and -e pattern3

The above command will print lines matching all the patterns at once.

--no-index Search files in the current directory that is not managed by Git.

Check man git-grep for help.

See also:

For OR operation, see:

2
  • This worked the best for me
    – user582175
    Commented Mar 25, 2021 at 17:37
  • 4
    This will only match a file if a line is found that matches all patterns, not files where the patterns match individual lines. Use --all-match and --or instead of --and for that. Commented Jan 26, 2023 at 13:05
18

grep pattern1 | grep pattern2 | ...

I would like to use single grep because I am building arguments dynamically, so everything has to fit in one string

It's actually possible to build the pipeline dynamically (without resorting to eval):

# Executes: grep "$1" | grep "$2" | grep "$3" | ...
function chained-grep {
    local pattern="$1"
    if [[ -z "$pattern" ]]; then
        cat
        return
    fi    

    shift
    grep -- "$pattern" | chained-grep "$@"
}

cat something | chained-grep all patterns must match order but matter dont

It's probably not a very efficient solution though.

4
  • 2
    Use either chained-grep() or function chained-grep but not function chained-grep(): unix.stackexchange.com/questions/73750/…
    – nisetama
    Commented Jan 19, 2019 at 17:08
  • Can you describe what the trick is? Can you add it to the answer (without "Edit:", "Update:", or similar ) by editing it? Commented Oct 30, 2020 at 20:40
  • Reformulated the answer to make the trick clearer (ie.: build a shell pipeline dynamically)
    – olejorgenb
    Commented Oct 30, 2020 at 23:21
  • 1
    The important part here is that shell allows recursion which makes this possible. Note the keyword local in front of variable that must be unique for the recursion. Also note that keyword local is not POSIX so using shebang #!/bin/sh may not be safe, see details here: unix.stackexchange.com/a/493743/20336 Commented Jul 7, 2022 at 7:15
12
+50

If patterns contains one pattern per line, you can do something like this:

awk 'NR==FNR{a[$0];next}{for(i in a)if($0!~i)next}1' patterns -

Or this matches substrings instead of regular expressions:

awk 'NR==FNR{a[$0];next}{for(i in a)if(!index($0,i))next}1' patterns -

To print all instead of no lines of the input in the case that patterns is empty, replace NR==FNR with FILENAME==ARGV[1], or with ARGIND==1 in gawk.

These functions print the lines of STDIN which contain each string specified as an argument as a substring. ga stands for grep all and gai ignores case.

ga(){ awk 'FILENAME==ARGV[1]{a[$0];next}{for(i in a)if(!index($0,i))next}1' <(printf %s\\n "$@") -; }
gai(){ awk 'FILENAME==ARGV[1]{a[tolower($0)];next}{for(i in a)if(!index(tolower($0),i))next}1' <(printf %s\\n "$@") -; }
1
  • crisp answer which addresss several uses cases and works (verified on macos) Commented Aug 8, 2020 at 18:19
4

Here's my take, and this works for words in multiple lines:

Use find . -type f followed by as many
-exec grep -q 'first_word' {} \;
and the last keyword with
-exec grep -l 'nth_word' {} \;

-q quiet / silent
-l show files with matches

The following returns list of filenames with words 'rabbit' and 'hole' in them:
find . -type f -exec grep -q 'rabbit' {} \; -exec grep -l 'hole' {} \;

1
  • If you look carefully, you just might learn that this is not the functionality that the question is asking for. Commented Dec 7, 2020 at 5:40
2

to search multiple files for the presence of two patterns anywhere in the file use

awk -v RS="" '/patern1/&&/patern2/{print FILENAME}' file1 ... filen
2
  • Grep is all too often used where (IMO) awk would be better. I like this answer for exactly that reason, and of course awk can do further processing such as printing only fields 6 and 2 from the input. Commented Jun 1, 2021 at 10:20
  • This doesn't actually address the OP's question, but +1 'cause I think it's very useful for other related situations & reveals the strength of awk... if you had to choose awk or grep, I think it's clear. Fortunately we don't have to make this choice :)
    – Seamus
    Commented Feb 19, 2022 at 0:54
1

ripgrep

Here is the example using rg:

rg -N '(?P<p1>.*pattern1.*)(?P<p2>.*pattern2.*)(?P<p3>.*pattern3.*)' file.txt

It's one of the quickest grepping tools, since it's built on top of Rust's regex engine which uses finite automata, SIMD and aggressive literal optimizations to make searching very fast.

See also related feature request at GH-875.

0

just directly MULTIPLY the patterns if you want them all to be true, thus eliminating any and all conditional branching

awk '/regexp1/ * /regexp2/ * /regexp3/ … '

say if you need regex 4 FALSE while regex 5/6 both being TRUE, then you can lump them all into a single compare :

awk '/regexp4/ < /regexp5/ * /regexp6/'

or say if you want to match either regex 7 or regex 8, but not both at the same time, then do either one of these

logical "!=" NOT EQUAL

awk '/regexp7/ != /regexp8/'

arithmetic "-" MINUS, since [ A XOR B ] on a single-bit level 
is same as checking for non-zero result of subtraction

awk '/regexp7/ - /regexp8/'

A real world example of this particular combination would be for checking whether a certain month has 31 days or not :

jot 12 | awk '(_ = +$1) % 2 != (7 < _)'
                                        or                   
         awk '((_ = +$1) + (7 < _)) % 2'

     1
     3
     5
     7
     8
    10
    12

Conversely, checking for a month being short would be :

jot 12 | awk  '(_ = +$1) % 2 == (7 < _)'
         awk  '(_ = +$1) % 2 -  (_ < 8)'                               
         awk '((_ = +$1)     +  (_ < 8)) % 2'

     2
     4
     6
     9
    11

here's the strangest one of them all - if you want either regex 9 to be TRUE or regex 10 to be FALSE, and wanna do it without conditional branching :

awk '/regexp9/ ^ /regexp10/'

That's right - regex 9 RAISED TO THE POWER of regex 10. It works because

    1 1 1^1 ->  1
    1 0 1^0 ->  1
    0 1 0^1 -> [0]
    0 0 0^0 ->  1

So the only time this algebraic expressions yields FALSE would be when regex 9 is FALSE while regex 10 is TRUE. Its twin via logical comparison operators would be :

awk '/regexp9/ >= /regexp10/'

All these might not appear to be idiomatic, but they're all POSIX-compliant awk syntax that is fully portable.

0

Whereas not as elegant or fast, it is easy to remember. For two patterns:

grep B $(grep -l A *)

You can pipe a list of files (-l) that meet your first pattern to the next grep, for example:

➜ grep A *
a:A
ab:A

➜ grep B *
ab:B
b:B

➜ grep B $(grep -l A *)
ab:B

You can nest them as well, but you will need to add -l:

grep -l C $(grep -l B $(grep -l A *))
1
  • Will easily break for filenames containing spaces, asterisks, etc.
    – muru
    Commented Jul 4 at 12:44
-2

To find all of the words (or patterns), you can run grep in a for loop. The main advantage here is searching from a list of regular expressions.

A real example:

# File 'search_all_regex_and_error_if_missing.sh'

find_list="\
^a+$ \
^b+$ \
^h+$ \
^d+$ \
"

for item in $find_list; do
   if grep -E "$item" file_to_search_within.txt
   then
       echo "$item found in file."
   else
       echo "Error: $item not found in file. Exiting!"
       exit 1
   fi
done

Now let's run it on this file:

hhhhhhhhhh
aaaaaaa
bbbbbbbbb
ababbabaabbaaa
ccccccc
dsfsdf
bbbb
cccdd
aa
caa
$ ./search_all_regex_and_error_if_missing.sh
aaaaaaa aa
^a+$ found in file.
bbbbbbbbb bbbb
^b+$ found in file.
hhhhhhhhhh
^h+$ found in file.
Error: ^d+$ not found in file. Exiting!
10
  • 2
    Your logic is faulty -- I asked for ALL operator, your code works as OR operator, not AND. And btw. for that (OR) is much easier solution given right in the question. Commented Aug 14, 2018 at 22:18
  • @greenoldman The logic is simple: The for will loop on ALL of the words/patterns in the list, and if it is found in file - will print it. So just remove the else if you don't need action in case word was not found.
    – Noam Manos
    Commented Aug 16, 2018 at 15:07
  • 1
    I understand your logic as well as my question -- I was asking about AND operator, meaning the file is only a positive hit if it matches pattern A and pattern B and pattern C and... AND In you case file is positive hit if it matches pattern A or pattern B or... Do you see the difference now? Commented Aug 17, 2018 at 6:19
  • @greenoldman not sure why you think this loop does not check AND condition for all patterns? So I've edited my answer with a real example: It will search in file for all regex of list, and on the first one which is missing - will exit with error.
    – Noam Manos
    Commented Aug 19, 2018 at 15:04
  • 1
    Note that this answer is about searching for all patterns and reporting if each pattern cannot be find at least once in the file. The original question was about matching ALL the patterns against ALL the lines instead of matching files. Commented Jul 7, 2022 at 7:27

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .