4

I have a file file1.txt with the following data:

MIME_HTML_ONLY
SUSPICIOUS_RECIPS
SORTED_RECIPS
HFILTER_HELO_5

and another file file2.txt with lines as:

HFILTER_FROM_BOUNCE FORGED_OUTLOOK_HTML
SORTED_RECIPS HFILTER_HELO_5
MIME_HTML_ONLY HFILTER_FROM_BOUNCE
SUSPICIOUS_RECIPS ANY_OTHER_WORD
:
:

I want to find all the lines in file2.txt who's data (all the words of that line) is the subset of the data present in file1.txt

For e.g the output above should be the line : SORTED_RECIPS HFILTER_HELO_5

Now I can loop through and read individual lines of file2.txt and see if its the subset of file1.txt. But i've have to do this for say a 1000 different file1.txt's. So looping through individual lines of file2.txt for every file1.txtis very slow. Is there any efficient way to do this using awk sed grep ?

  • For 1000 different file1.txt's, do you want to know exactly which file1.txt is matching against file2.txt? or just want to know whether or not there's matching file1.txt? – yaegashi Aug 7 '15 at 0:54
  • Actually for any file1.txt, i just want to know if there's any line in file2.txt that's the subset of data in file1.txt(as mentioned above). I've to check this condition for all the other 1000 file1.txt's and exactly want to know the file, since i'm performing later operations with that file1.txt. So i'll probably loop through them which isn't the issue once i get to know the solution. – awhitesong Aug 7 '15 at 1:01
  • Ok. IMHO it would suffice to just loop over 1000 file1.txt's running the script in my answer, if file2.txt contains less than 1M lines. Otherwise it would be better to find way to compile all file1.txt's into one sed/awk script. – yaegashi Aug 7 '15 at 1:14
3

The following script is compiling file1.txt into a single regex for grep -E.

#!/bin/sh
regex="^($(awk '{printf $0"|"}' $1) )+\$"
grep -E "$regex" $2

Usage:

$ ./script.sh file1.txt file2.txt 
SORTED_RECIPS HFILTER_HELO_5

$regex is compiled from file1.txt as follows:

^(ME_HTML_ONLY|SUSPICIOUS_RECIPS|SORTED_RECIPS|HFILTER_HELO_5| )+$

For thousands of file1.txt and file2.txt with millions of lines, it would be better to compile all of file1.txt's into a single awk program with the following script:

#!/bin/sh
for i; do
        regex="^($(awk '{printf $0"|"}' $i) )+\$"
        echo "/$regex/ { print \"$i: \"\$0 }"
done

For example (file1.txt's are named as match1.txt match2.txt match3.txt):

$ ./script2.sh match*.txt 
/^(ME_HTML_ONLY|SUSPICIOUS_RECIPS|SORTED_RECIPS|HFILTER_HELO_5| )+$/ { print "match1.txt: "$0 }
/^(HFILTER_FROM_BOUNCE|FORGED_OUTLOOK_HTML|ANY_OTHER_WORD| )+$/ { print "match2.txt: "$0 }
/^(SORTED_RECIPS|HFILTER_HELO_5|MIME_HTML_ONLY|HFILTER_FROM_BOUNCE| )+$/ { print "match3.txt: "$0 }

$ ./script2.sh match*.txt >match.awk
$ awk -f match.awk file2.txt 
match2.txt: HFILTER_FROM_BOUNCE FORGED_OUTLOOK_HTML
match1.txt: SORTED_RECIPS HFILTER_HELO_5
match3.txt: SORTED_RECIPS HFILTER_HELO_5
match3.txt: MIME_HTML_ONLY HFILTER_FROM_BOUNCE
  • That is indeed a great converting the contents of file into a regex. Totally worked. I think looping through the file would suffice. Sucks that i can't upvote. Anyway, thanks. – awhitesong Aug 7 '15 at 1:23
  • @awhitesong Updated, just for the case you have file2.txt with millions of lines. – yaegashi Aug 7 '15 at 1:37
4
awk 'FNR == NR && $0 !~ /^[[:blank:]]*$/ { Dict[$0] = 1 }
     FNR != NR {
        i = 1
        while( i <= NF && Dict[ $i] == 1) i++
        if( i > NF) print
        }
    ' File1.txt File2.txt
  • generic, non dependant to file2 number of field/word per line
  • work with sorted and unsorted content of both file
  • use memory to load first file in Dictionnary so maybe not the best if huge number of word to validate
  • files order to provide to awk is mandatory
    • 1st is dictionnary reference
    • any (at least 1) other are file to filter

Concept:

  • load each word in a array using the value as index
    • taking 1 as value (unassigned have 0 by default)
    • from first file [where FNR (file record number) = NR (record number since first opened file) and a record is, by default, a line in awk]
    • there is a filter on empty line (no character or only space)
  • initialise a counter (i)
  • compare each field (word here due to space separator by default) to his equivalent in Dictionnary. If exist (value = 1), loop to next field and increment a counter (i)
  • after the loop, if the counter (i) is bigger than the number of field (word), all word are matching, we print the line
  • loop to next line entry

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.