Script comparing two files, match two strings anywhere on line

Question

I am trying to find an efficient way to do some comparisons using a bash script. I am using grep and awk, but it is very slow and I don't have a good way to separate the matches.

Consider input file1:

311 2222
833 7777
Jam 33333

and input file2:

A 833 0 0 0 0 7777 0 0 0 0
B no match - 2222 833 3333
C the cow jumps over the 311 moon 2222
D illicit Jam fox 33333 
E no match - Jam 
F 7777 833

In reality, file1 is ~100 lines and file2 is ~10,000 lines.

I want to compare the two strings in file1 with each line in file2. If both strings match anywhere on that line, print the line from file2. For the above example, ideally the output would be similar this (but formatted into columns):

C the cow jumps over the 311 moon 2222
---<separator>---
A 833 0 0 0 0 7777 0 0 0 0
F 7777 833
---<separator>--- 
D illicit Jam fox 33333 

Notice that matches are grouped together with separators between them (833 7777 matches two lines in file2). It is preferred that all matches are word matches.

One more note: each line in file1 will match one or more lines in file2, but each line in file2 will match either 1 or 0 lines in file1.

[updated "no match" lines in file2]

Thanks

Answer 1

This is a pure awk solution that works for two patterns per line (space delimited) in patterns file with logical AND operation:

awk 'NR==FNR{patts[$1]=$2;next}{for (i in patts) if (($0 ~ i) && ($0 ~ patts[i])) print}' patterns file

Update:
For word instead of regex matching you can evaluate this alternative :

awk 'NR==FNR{patts[$0]="\\<" $1 "\\>.*\\<" $2 "\\>|\\<" $2 "\\>.*\\<" $1 "\\>";next} \
{for (i in patts) if ($0 ~ patts[i]) print}' patterns file1   

It is actually transforming the pattern 833 7777 to
\<833\>.*\<7777\> | \<7777\>.*\<833\> which makes word matching and simulates also logical AND for the two patterns.

This solution is tested and does not match records like G 77771 2833

Update No2
This will ensure word matching, logical AND operation, and printing per matched groups with separator string.

awk 'NR==FNR{patts[$0]="\\<" $1 "\\>.*\\<" $2 "\\>|\\<" $2 "\\>.*\\<" $1 "\\>";next} \
{for (i in patts) {if ($0 ~ patts[i]) !found[i]?found[i]=$0:found[i]=found[i] ORS $0}} \
END{for (k in found) {print found[k];print "-----"}}' patterns file1

#Output
A 833 0 0 0 0 7777 0 0 0 0
F 7777 833
-----
D illicit Jam fox 33333
-----                  
C the cow jumps over the 311 moon 2222
----- 

Online Testing Here.

PS: Due to the awk way of working with associative arrays we can not affect in the END section the printing of the found array. It will be somehow "random".

Answer 2

If it is known that there are exactly two strings in each line of file1:

while read -ra elements; do
   grep "${elements[0]}" file2 | \
     grep "${elements[1]}" && \
     echo "----"
done < file1

Answer 3

I'd do it in perl because I think it looks clearer:

#!/usr/#bin/env perl
use strict;
use warnings;
use Data::Dumper;


my ( $pattern_file_name, $process_file_name ) = @ARGV; 

open ( my $patterns_file, '<', $pattern_file_name ) or die $!;
my @matches = map { [split] } <$patterns_file>;
close ( $patterns_file );

print "Using:\n";
print Dumper \@matches;

#my @matches = ( [ '311', '2222' ], [ '833', '7777' ], [ 'Jam', '33333' ] );

#read main file
my @results;
open  ( my $input, '<', $process_file_name ) or die $!; 

#iterate a line at a time. 
while ( my $line = <$input> ) {
 GROUP:
   for my $id ( 0 .. $#matches ) {
      #Check each set of expressions.
      foreach my $expression ( @{ $matches[$id] } ) { 
         #move to the next group if any don't match
         next GROUP unless $line =~ m/$expression/;
      }
      #didn't get skipped, so must have matched all. 
      push( @{ $results[$id] }, $line );
   }
}
print Dumper \@results;

print "\n$_\n" for @results;
close ( $input );

Answer 4

There is a trick you can make here that works fine in my tests:

$ awk -f <(sed 's/^\|$/\//g; s/ /\/ \&\& \//g' file1) file2
#Output
A 833 0 0 0 0 7777 0 0 0 0
C the cow jumps over the 311 moon 2222
D illicit Jam fox 33333
F 7777 833

The success of this solution is because we convert the file1 (patterns) in the awk format /pattern1/ && /pattern2/{print} (print is the default action and can be ommitted)

$ sed 's/^\|$/\//g; s/ /\/ \&\& \//g' file1
/311/ && /2222/
/833/ && /7777/
/Jam/ && /33333/

Instead of writting with sed a new file, we feed sed directly to awk as a file using process substitution.

So the command :
awk -f <(sed 's/^\|$/\//g; s/ /\/ \&\& \//g' file1) file2 instructs awk to read the programm from the process substition and apply it to file2.

Mind that i have skipped the separators, since is not clear where exactly you need them.

PS: It should be offcourse a pure awk solution.

Answer 5

perl -wMstrict -Mvars='*f2' -l -0777ane '
   if ( ! @ARGV ) {# this is File1 zone: slurped in $_
      while ( /^(\S+)\s+(\S+)$/mg ) {
         my $rx = qr/^(?=.*$1)(?=.*$2)/m; # AND matching of $1/$2
         pos($f2)=0;
         $f2 =~ /\G([^\n]+)/m and print $1 while $f2 =~ /$rx/mg;
         print "--- <Separator> ---" unless /\G\n\z/;
      }
   } else {# This is File2 zone: slurped whole in $f2
      $f2 = $_;
   }
' File2 File1 #<----- order is important here

Explanation:

We have to keep in mind the order of printing is very important here. The lines of File2 are printed from the order determined by the strings to be found in File1. And after every line of File1 is pattern matched against File1 we need a separato line as well, irrespective of whether the match succeeded or not. The files are slurped, File2 => $f2, and File is treated as $_ in