0

I have a file of patterns which I want to compare to a directory of files.

The pattern file contents looks something like this (and could be regular expressions):

pattern-that-occurs-in-file
pattern-that-also-occurs-in-file

Example search files that should come up if their contents match the patterns:

unrelated content
pattern-that-occurs-in-file
more unrelated content
pattern-that-also-occurs-in-file
further unrelated content

Or:

unrelated content
pattern-that-also-occurs-in-file
more unrelated content
pattern-that-occurs-in-file
further unrelated content

Example search files that should not come up:

unrelated content
more unrelated content
pattern-that-occurs-in-file
further unrelated content

Or:

unrelated content
pattern-that-also-occurs-in-file
more unrelated content
further unrelated content

Or:

unrelated content
more unrelated content
further unrelated content

I need grep to output a list of files where both patterns occur. I don't care if I can see the matching lines or not.

Can I do this in a single command? If so, how?

1
  • You say "pattern", but the strings that you show says "string". Are they regular expressions or strings that you want to match exactly? Do I understand you correctly that you require all strings to match for a file to be reported?
    – Kusalananda
    Dec 3, 2019 at 17:20

4 Answers 4

1

Not exactly a single command, but:

num_patterns=$( wc -l < patterns_file )
for file in dir/*; do
    num_occurrances=$( grep -F -o -f patterns_file "$file" | sort -u | wc -l )
    if (( num_patterns == num_occurrances )); then
        echo "all patterns in $file"
    fi
done

This approach won't work when the patterns are regular expressions, because the matching text probably won't be unique for every match.

4
  • I believe that this may possibly give the wrong result if a pattern matches multiple times in a file.
    – Kusalananda
    Dec 3, 2019 at 18:53
  • @Kusalananda the sort -u should prevent that
    – roaima
    Dec 3, 2019 at 19:31
  • 1
    @roaima Together with grep -o, yes, you may be right.
    – Kusalananda
    Dec 3, 2019 at 19:32
  • 1
    Here are two examples where this algorithms fails. First, pattern 1. matches "1a" and "1b" in a file, and pattern 2. matches nothing, but your algorithm will conclude 2 patterns = 2 occurrences. Second, pattern 123 matches "1234", but then pattern 12, which does match, won't produce any additional output. Only "123" will be output, and your algorithm will conclude 2 patterns != 1 occurrence.
    – webb
    Mar 25 at 0:32
1

Assuming that ./*.txt matches all the files that you are interested in, and that you want to find the files that contains all of the strings in the file ./patterns (may contain more than two lines):

#!/bin/bash

pathnames=( ./*.txt )

while IFS= read -r pattern; do
    for pathname in "${pathnames[@]}"; do
        pathnames=( ${pathnames[@]:1} )

        if grep -qF -e "$pattern" "$pathname"; then
            pathnames+=( "$pathname" )
        fi
    done
done < ./patterns

printf 'Matched: %s\n' "${pathnames[@]}"

This loops over the patterns. For each pattern, it tests it against all files in the array pathnames. If the pattern matches, we keep the current pathname in the array, otherwise it's thrown away. At the end, pathnames will contain only the pathnames that contain all patterns.

Because of the way the pathnames array is managed, the number of grep calls made for each pattern will decrease as more and more files are discarded.

The command pathnames=( ${pathnames[@]:1} ) will shift off the first (current) pathname off from the array, while pathnames+=( "$pathname" ) puts it back in again at the end.

The command grep -qF -e "$pattern" "$pathname" will return a true value if the file $pathname contains the string in $pattern. We use -q to make grep quiet and also to make it exit as soon as it matches the pattern in the file. We use -F to do string comparisons rather than regular expression matches.


Only because I like the terser sh syntax more than named arrays in bash, here's a variation of the above for /bin/sh (the positional parameters replaces the pathnames array):

#!/bin/sh

set -- ./*.txt

while IFS= read -r pattern; do
    for pathname do
        shift

        if grep -qF -e "$pattern" "$pathname"; then
            set -- "$@" "$pathname"
        fi
    done
done < ./patterns

printf 'Matched: %s\n' "$@"
1
  • Dear everyone: the other two answers give incorrect results. Please use this textbook correct solution instead.
    – webb
    Mar 25 at 0:37
0

If I understand you correctly, this could be an option (if my logic is sound). Here I assume that the patterns are unique on each file:

grep -R < file_with_patterns . | cut -d':' -f1 | uniq -d

grep will return two lines if the two patterns match, or just one or none. Taking advantage of this situation, we use uniq -d to show only duplicate results of file names.

0

@glenn-jackman's and @schrodigerscatcuriosity's answers fail regular expressions (OP modified the question to also include regular expressions). For example, pattern 1. matches "1a" and "1b" in a file, and pattern 2. matches nothing, but both algorithms conclude the file was matched by both patterns. Second, pattern 123 matches "1234", but then pattern 12, which does match, won't cause grep to produce any additional output. Both algorithms will conclude that the file only matches one of the two patterns.

@kusalananda's works well, but a more efficient solution may be possible:

files=`find ./*.txt`
while read pattern; do
    files=`echo "$files" | xargs grep -l "$pattern"` || break
done < ./patterns
echo Matched: $files

This solution is similar to @kusalananda's: it loops over the patterns, removing any files that don't match. However, this solution uses xargs grep -l instead of a nested loop for the files. It therefore runs about one grep process per pattern instead of one grep process per pattern per file, so it should be an order of magnitude faster.

PS: This solution doesn't handle spaces in filenames, whereas @kusalananda's does. But this solution can be easily modified to handle spaces in filenames. If you have spaces or other bad characters in your filenames, then first, hang your head in shame, and second, change

xargs

to

tr \\n \\0 | xargs -0

I did not put this as the main solution because it looks confusing and is not relevant to the main issue.

PPS: For maximum speed, put the most rare patterns first in the patterns file, and the more common patterns last in order to eliminate as many files as possible as early as possible.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.