2

I am trying to find an efficient way to do some comparisons using a bash script. I am using grep and awk, but it is very slow and I don't have a good way to separate the matches.

Consider input file1:

311 2222
833 7777
Jam 33333

and input file2:

A 833 0 0 0 0 7777 0 0 0 0
B no match - 2222 833 3333
C the cow jumps over the 311 moon 2222
D illicit Jam fox 33333 
E no match - Jam 
F 7777 833

In reality, file1 is ~100 lines and file2 is ~10,000 lines.

I want to compare the two strings in file1 with each line in file2. If both strings match anywhere on that line, print the line from file2. For the above example, ideally the output would be similar this (but formatted into columns):

C the cow jumps over the 311 moon 2222
---<separator>---
A 833 0 0 0 0 7777 0 0 0 0
F 7777 833
---<separator>--- 
D illicit Jam fox 33333 

Notice that matches are grouped together with separators between them (833 7777 matches two lines in file2). It is preferred that all matches are word matches.

One more note: each line in file1 will match one or more lines in file2, but each line in file2 will match either 1 or 0 lines in file1.

[updated "no match" lines in file2]

Thanks

  • @don_crissti - only word matches are desired. – mstream Mar 9 '17 at 23:11
  • Please edit your post and specify that you only want to print on word match (or string equality) – don_crissti Mar 9 '17 at 23:39
1

This is a pure awk solution that works for two patterns per line (space delimited) in patterns file with logical AND operation:

awk 'NR==FNR{patts[$1]=$2;next}{for (i in patts) if (($0 ~ i) && ($0 ~ patts[i])) print}' patterns file

Update:
For word instead of regex matching you can evaluate this alternative :

awk 'NR==FNR{patts[$0]="\\<" $1 "\\>.*\\<" $2 "\\>|\\<" $2 "\\>.*\\<" $1 "\\>";next} \
{for (i in patts) if ($0 ~ patts[i]) print}' patterns file1   

It is actually transforming the pattern 833 7777 to
\<833\>.*\<7777\> | \<7777\>.*\<833\> which makes word matching and simulates also logical AND for the two patterns.

This solution is tested and does not match records like G 77771 2833

Update No2
This will ensure word matching, logical AND operation, and printing per matched groups with separator string.

awk 'NR==FNR{patts[$0]="\\<" $1 "\\>.*\\<" $2 "\\>|\\<" $2 "\\>.*\\<" $1 "\\>";next} \
{for (i in patts) {if ($0 ~ patts[i]) !found[i]?found[i]=$0:found[i]=found[i] ORS $0}} \
END{for (k in found) {print found[k];print "-----"}}' patterns file1

#Output
A 833 0 0 0 0 7777 0 0 0 0
F 7777 833
-----
D illicit Jam fox 33333
-----                  
C the cow jumps over the 311 moon 2222
----- 

Online Testing Here.

PS: Due to the awk way of working with associative arrays we can not affect in the END section the printing of the found array. It will be somehow "random".

  • @don_crissti By the way, even the grep solution by DopeGhoti does not handle word matching or correct separation , right? It is a classic grep , meaning that your "G" record will be matched. – George Vasiliou Mar 10 '17 at 13:51
  • Yes, I personally try to avoid using regex, I would have used field comparison (so iterate over fields in file2 and compare if they're equal to those in file1 using a counter to check when two matches on the same line) - that would prolly mean adding some variable to account for all that but anyway +1 for the effort – don_crissti Mar 10 '17 at 14:11
  • @don_crissti Thanks. Your approach sounds interesting. If you could set a quick snippet up would be nice. PS: I also corrected the extra blank line issue in grouping using a small ternary if. – George Vasiliou Mar 10 '17 at 14:18
  • @mstream When you have the time, would be nice to know if update 2 solution worked for you as expected. – George Vasiliou Mar 10 '17 at 14:21
  • This could be done awk-only but I'm too lazy: awk 'NR==FNR{z[NR]=$0;next}{a=0; b=0; for (i=1; i<=length(z); i++){split(z[i], y); for (j=1; j<=NF; j++){if ($j==y[1]){a=1} else if ($j==y[2]){b=1}};if (a && b){print i, FNR, $0};a=0;b=0}}' file1 file2 | sort -t' ' -k1,2n | awk 'NR>1{if (!seen[$1]++){print "- - ----------"}}1' | cut -d' ' -f3- – don_crissti Mar 10 '17 at 14:26
1

If it is known that there are exactly two strings in each line of file1:

while read -ra elements; do
   grep "${elements[0]}" file2 | \
     grep "${elements[1]}" && \
     echo "----"
done < file1
  • 1
    Won't that match any lines in file2 with at least one of the strings in file1, not just lines with a matching pair? – ilkkachu Mar 9 '17 at 22:30
  • Sorry, I was in a sed state of mind. Re-revised. – DopeGhoti Mar 9 '17 at 23:01
  • @DopeGhoti - Thanks for the help. I gave it a try, and it's giving me way too many matched lines. I'll take another look tomorrow to see if I messed something up. – mstream Mar 9 '17 at 23:27
0

There is a trick you can make here that works fine in my tests:

$ awk -f <(sed 's/^\|$/\//g; s/ /\/ \&\& \//g' file1) file2
#Output
A 833 0 0 0 0 7777 0 0 0 0
C the cow jumps over the 311 moon 2222
D illicit Jam fox 33333
F 7777 833

The success of this solution is because we convert the file1 (patterns) in the awk format /pattern1/ && /pattern2/{print} (print is the default action and can be ommitted)

$ sed 's/^\|$/\//g; s/ /\/ \&\& \//g' file1
/311/ && /2222/
/833/ && /7777/
/Jam/ && /33333/

Instead of writting with sed a new file, we feed sed directly to awk as a file using process substitution.

So the command :
awk -f <(sed 's/^\|$/\//g; s/ /\/ \&\& \//g' file1) file2 instructs awk to read the programm from the process substition and apply it to file2.

Mind that i have skipped the separators, since is not clear where exactly you need them.

PS: It should be offcourse a pure awk solution.

0
perl -wMstrict -Mvars='*f2' -l -0777ane '
   if ( ! @ARGV ) {# this is File1 zone: slurped in $_
      while ( /^(\S+)\s+(\S+)$/mg ) {
         my $rx = qr/^(?=.*$1)(?=.*$2)/m; # AND matching of $1/$2
         pos($f2)=0;
         $f2 =~ /\G([^\n]+)/m and print $1 while $f2 =~ /$rx/mg;
         print "--- <Separator> ---" unless /\G\n\z/;
      }
   } else {# This is File2 zone: slurped whole in $f2
      $f2 = $_;
   }
' File2 File1 #<----- order is important here

Explanation:

We have to keep in mind the order of printing is very important here. The lines of File2 are printed from the order determined by the strings to be found in File1. And after every line of File1 is pattern matched against File1 we need a separato line as well, irrespective of whether the match succeeded or not. The files are slurped, File2 => $f2, and File is treated as $_ in

1

I'd do it in perl because I think it looks clearer:

#!/usr/#bin/env perl
use strict;
use warnings;
use Data::Dumper;


my ( $pattern_file_name, $process_file_name ) = @ARGV; 

open ( my $patterns_file, '<', $pattern_file_name ) or die $!;
my @matches = map { [split] } <$patterns_file>;
close ( $patterns_file );

print "Using:\n";
print Dumper \@matches;

#my @matches = ( [ '311', '2222' ], [ '833', '7777' ], [ 'Jam', '33333' ] );

#read main file
my @results;
open  ( my $input, '<', $process_file_name ) or die $!; 

#iterate a line at a time. 
while ( my $line = <$input> ) {
 GROUP:
   for my $id ( 0 .. $#matches ) {
      #Check each set of expressions.
      foreach my $expression ( @{ $matches[$id] } ) { 
         #move to the next group if any don't match
         next GROUP unless $line =~ m/$expression/;
      }
      #didn't get skipped, so must have matched all. 
      push( @{ $results[$id] }, $line );
   }
}
print Dumper \@results;

print "\n$_\n" for @results;
close ( $input );
  • Trivial to read filenames out of @ARGV. Let me edit. Or you can use GetOpt to do a more fully featured 'argument' list. – Sobrique Mar 10 '17 at 14:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.