1

I have n separate non fixed greps against m files. I only need to know if there is at least 1 match in each file, but I need this for each pattern. I currently make n separate greps so I can merge them all later, but it is very slow and some of the files are large.

Is there a way to replace these that doesn't need me to read all the files n times (doesn't need to be separate files as long as I can match the pattern (not match) to the files that contain a match). grep -f looked promising but it shows the files that match any pattern instead of the files that match each pattern.

ex of what gets merged later into 1 large file:

grep -liE  pattern1  file_glob* > temp_pattern1.txt && sed s/^/escapedpattern1 / temp_pattern1.txt
grep -liE  pattern2   file_glob* > temp_pattern2.txt && sed s/^/escapedpattern2 / temp_pattern2.txt
...
grep -liE  patternN   file_glob* > temp_patternN.txt && sed s/^/escapedpatternN / temp_patternN.txt

temp_pattern1.txt
pattern1 /path/to/file1
pattern1 /path/to/file2
pattern1 /path/to/file3

temp_pattern2.txt
pattern2 /path/to/file1
pattern2 /path/to/file3
...
temp_patternN.txt
pattern N /path/to/fileM

1 Answer 1

1

If you want to use grep, the best you can do is use the -m 1 option to make grep stop reading its current input file on the first match. You'll still be reading each input file multiple times (once for each pattern), but it should be faster (unless a match is on or near the last line of the file).

e.g.

#!/bin/bash

# Speed up each grep by exiting on 1st match with -m 1
#
# This still reads each file multiple times, but should run faster because it
# won't read the entire file each time unless the match is on the last line.
#
# Also reduce repetitive code by using an array and a for loop iterating over
# the indices of the array, rather than the values

patterns=(pattern1 pattern2 pattern3 patternN)

# iterate over the indices of the array (with `${!`), not the values.
for p in "${!patterns[@]}"; do
  # escape forward- and back- slashes in pattern
  esc=$(echo "${patterns[$p]}" | sed -e 's:/:\\/:g; s:\\:\\\\:g')
  grep -liE -m 1 "${patterns[$p]}" file_glob* |
    sed -e "s/^/$esc\t/" > "temp_pattern$(($p+1)).txt"
done

Note: the $p+1 is there because bash arrays start from zero. the +1 makes the temp_patterns files start from 1.


You can do what you want if you use a scripting language like awk or perl. e.g. the following perl script reads each input file only once and checks each line against each pattern that hasn't been seen in that file yet. It keeps track of patterns that have already been seen in a particular file (using the @seen array) and also notices when all available patterns have been seen in a file (also using @seen) and closing the current file in that case.

#!/usr/bin/perl
use strict;

# array to hold the patterns
my @patterns = qw(pattern1 pattern2 pattern3 patternN);

# Array-of-Arrays (AoA, see man pages for perllol and perldsc)
# to hold matches
my @matches;

# Array for keeping track of whether current pattern has
# been seen already in current file
my @seen;

# read each line of each file
while(<>) {
  # check each line against all patterns that haven't been seen yet
  for my $i (keys @patterns) {
    next if $seen[$i];
    if (m/$patterns[$i]/i) {
      # add the pattern and the filename to the @matches AoA
      push @{ $matches[$i] }, "$patterns[$i]\t$ARGV";
      $seen[$i] = 1;
    }
  };

  # handle end-of-file AND having seen all patterns in a file
  if (eof || $#seen == $#patterns) {
    #print "closing $ARGV on line $.\n" unless eof;
    # close the current input file.  This will have
    # the effect of skipping to the next file.
    close(ARGV);
    # reset seen array at the end of every input file
    @seen = ();
  };
}

# now create output files
for my $i (keys @patterns) {
  #next unless @{ $matches[$i] }; # skip patterns with no matches
  my $outfile = "temp_pattern" . ($i+1) . ".txt";
  open(my $out,">",$outfile) || die "Couldn't open output file '$outfile' for write: $!\n";
  print $out join("\n", @{ $matches[$i] }), "\n";
  close($out);
}

The if (eof || $#seen == $#patterns) line tests for either eof (end of file) on the current file OR if we have already seen all available patterns in the current file (i.e. if the number of elements in @seen is equal to the number of elements in @patterns).

In both cases, we want to reset the @seen array to empty, to be ready for the next input file.

And in the latter case, we also want to close the current input file early - we've already seen everything we want to see in it, there's no need to keep reading and processing the rest of the file.

BTW, if you don't want empty files to be created (i.e. when there is no match for a pattern), uncomment the next unless @{ $matches[$i] } line in the output for loop.


If you don't need or want the the temporary files, and want to just output all the matches to one file, replace the final output for loop with:

for my $i (keys @patterns) {
  #next unless @{ $matches[$i] }; # skip patterns with no matches
  print join("\n", @{ $matches[$i] }), "\n";
}

and redirect the output to a file.


BTW, if you want to add the line number that a pattern was first seen in a file, change:

push @{ $matches[$i] }, "$patterns[$i]\t$ARGV";

to

push @{ $matches[$i] }, "$patterns[$i]\t$.\t$ARGV";

$. is a built-in perl variable that holds the current line number of the input <>. It is reset to zero whenever the current file (ARGV) is closed.

1
  • Re your first sentence: grep will stop at the first match if -l is used, which the user is doing already.
    – Kusalananda
    Nov 17, 2022 at 7:58

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .