4

I have a file with this content:

$ cat compromised_header.txt
some unique string 1
some other unique string 2
another unique string 3

I wanted to find all files that have all the lines of above file exactly in the same order and those lines have no intermediary lines in between.

Example input file:

$ cat a-compromised-file.txt
some unique string 1
some other unique string 2
another unique string 3
unrelated line x
unrelated line y
unrelated line z

I tried using below grep:

grep -rlf compromised_header.txt dir/

But I wasn't sure it will give the expected files as it will also match this file:

some unique string 1
unrelated line x
unrelated line y
unrelated line z
6
  • Are the lines that you're looking for always supposed to be at the start of an input file (lines 1-3), or can they be anywhere?
    – rowboat
    Apr 5, 2021 at 6:51
  • 1
    @rowboat not really, but they're always in the same exact order.
    – annahri
    Apr 5, 2021 at 7:08
  • 1
    in the same order but can they have intermediary lines in between? Apr 5, 2021 at 15:08
  • 2
    You should include some lines that contain regexp metachars and substrings in your sample input/output as you're getting some answers that will work with the example you posted but fail with different input. See how-do-i-find-the-text-that-matches-a-pattern for more information on that issue.
    – Ed Morton
    Apr 5, 2021 at 17:23
  • 2
    @OlivierDulac No, they have no intermediary lines in between.
    – annahri
    Apr 6, 2021 at 4:19

8 Answers 8

8

Using an awk that supports nextfile:

NR == FNR {
  a[++n]=$0; next
}
$0 != a[c+1] && (--c || $0!=a[c+1]) {
  c=0; next
}
++c >= n {
  print FILENAME; c=0; nextfile
}

with find for recursion:

find dir -type f -exec gawk -f above.awk compromised_header.txt {} +

Or this might work:

pcregrep -rxlM "$( perl -lpe '$_=quotemeta' compromised_header.txt )" dir

Using perl to escape metacharacters because pcregrep doesn't seem to combine --fixed-strings with --multiline.

With perl in slurp mode (won't work with files that are too large to hold in memory):

find dir -type f -exec perl -n0777E 'BEGIN {$f=<>} say $ARGV if /^\Q$f/m
' compromised_header.txt {} +
5

You'll need to use something more capable than grep, which can only do single-line matches.

perl, which can do multi-line matches, is perfect for this kind of job, combined with find to generate the list of files to search.

find dir/ -type f -iname '*.txt' -exec perl -e '
    local $/;    # slurp in entire files, instead of one line at a time

    my $firstfile = shift @ARGV;         # get name of the first file
    open(F,"<",$firstfile) or die "Error opening $firstfile: $!";
    my $first = <F>;                     # read it in
    close(F);
    my $search = qr/\Q$first\E/;         # compile to a fixed-string RE

    # now read in remaining files and see if they match
    while(<>) {
      next if ($ARGV eq $firstfile);
      if (m/$search/m) {
        print $ARGV,"\n";
      };
    }' ./compromised_header.txt {} +

This prints the filenames of any *.txt files in or below dir/ that contain the exact text in the first file ("compromised_header.txt").

Notes:

  • the qr// operator compiles a regex. The main use for this is to pre-compile an RE before using it in a loop, so that it doesn't waste time and cpu cycles getting re-compiled on every pass through the loop.

  • the \Q and \E used in the qr// operation mark the beginning and end of text in an RE pattern that is meant to be interpreted as a fixed string - i.e. all meta-characters that might be in the string will be quoted to disable their special meaning. See man perlre and search for "Quoting metacharacters" and perldoc -f quotemeta for details.

If that seems like an ugly, complicated, unreadable one-liner then try it like this, as a standalone script:

#!/usr/bin/perl

local $/;    # slurp in entire files, instead of one line at a time

my $firstfile = shift @ARGV;         # get name of the first file
open(F,"<",$firstfile) or die "Error opening $firstfile: $!";
my $first = <F>;                     # read it in
close(F);
my $search = qr/\Q$first\E/;         # compile to a fixed-string RE

# now read in remaining files and see if they match
while(<>) {
  next if ($ARGV eq $firstfile);
  if (m/$search/m) {
    print $ARGV,"\n";
  };
}

Save this as, e.g., check.pl and make it executable with chmod +x check.pl. Then run:

find dir/ -type f -iname '*.txt' \
  -exec ./check.pl ./compromised_header.txt {} +
10
  • 1
    This could be made slightly shorter by using the File::Slurp module, but that isn't guaranteed to be installed. Could also eliminate the need to run find too, by using the File::Find module (which is a standard part of any modern perl installation), but find works well enough, and there's no need to re-invent the wheel.
    – cas
    Apr 5, 2021 at 4:15
  • 1
    If you can read & write awk & sed, then perl is no harder. some people complain about perl looking like "line noise" and being "write-only", but that's because they think that regular expressions should be used with function calls rather than as operators (strangely, they don't make the same complaint about sed, which also has regex operators instead of functions).
    – cas
    Apr 5, 2021 at 7:28
  • 1
    That's not why people find perl scripts to be incomprehensible. If it were then they would have the same problem with sed as you mention. The language bloat caused by the focus on brevity over clarify is the issue.
    – Ed Morton
    Apr 5, 2021 at 17:34
  • 2
    @EdMorton perl doesn't have a "focus on brevity over clarity". It has a focus on expressivity, on being able to write and format your code in whichever style suits you - in general, or at the moment - with many different ways of doing anything. Perl's motto has long been "There's More Than One Way To Do It", and for good reason. Nobody, for example, could accuse my perl script above of brevity. I could have golfed it down to a line or two, but I like to write code that is readable and understandable, especially when I'm writing an example for others.
    – cas
    Apr 6, 2021 at 0:52
  • 1
    For example you use a construct m/$search/m. I've never seen anything like that in any other Unix tool so I'd like to know what it means. Maybe it has something to do with word boundaries? Maybe escaping regexp metachars? Whatever it is I'm sure there's a slightly lengthier way to do the same thing using constructs that are clear, obvious and common in other tools/languages. I've been trying for an hour to find some kind of documentation about it and have now given up. THAT is what I mean about the language having a focus on brevity over clarity - brief, arcane constructs for trivial tasks.
    – Ed Morton
    Apr 6, 2021 at 17:04
4

If you have GNU grep with PCRE -P mode then you operate in slurp mode -z and recursively -r list -l files matching the regex $re . The regex is constructed from the reference header file and escaping all characters special in a regex context for Perl.

re=$(< compromised_header.txt perl -lpe '$_=quotemeta')
re=${re//[${IFS#??}]/\\n}
grep -lrzP "(?m)^$re" .
4
$ cat tst.awk
NR==FNR {
    lines[++numLines] = $0
    next
}
FNR == 1 {
    expected = lines[++lineNr]
}
$0 == expected {
    if ( lineNr == numLines ) {
        print FILENAME
        found = 1
        exit
    }
    expected = lines[++lineNr]
}
END {
    exit !found
}

$ awk -f tst.awk compromised_header.txt 'a-compromised-file.txt'
a-compromised-file.txt

$ echo $?
0

Put the above inside a find to run it on subdirectories:

find dir -type f -exec awk -f tst.awk compromised_header.txt {} \;

Note that you need to use \; rather than + at the end of the find command so that a) it'll work with every version of find and every version of awk, and b) awk gets called 1 file at a time as it needs that to set the variables and exit with the correct status.

0
3

Assuming your search string doesn't have multiple trailing newlines or ASCII NUL characters (see pitfalls of reading file into shell variable for details) and you are okay with using ripgrep:

rg -lUF "$(< compromised_header.txt)" dir/

-F option is used so that file content is searched literally instead of being treated as a regexp

-U option enables multiline searching

rg will search recursively by default, however it also does smart filtering by default (respects .gitignore rules, ignores hidden files/folders, ignores binary files, etc). Use -uuu to make it behave like grep -r.


See my blog post Multiline fixed string search and replace with cli tools for more such multiline operations.

1
  • note: pcre2grep will also work.
    – user380915
    Apr 6, 2021 at 6:32
2
GNU grep:
grep -lzFf compromised_header.txt -r dir/
  • -z => "lines" ended by null byte instead of newline
  • -F => pattern is a fixed string, not a regex
  • -f file => patterns are read from file

Criticism in comments are well-justified: I didn't test thoroughly enough.

Another attempt is more complex: it replaces newlines in the pattern file and search files with an uncommon ASCII character:

find . -type f -print0 \
| xargs -0 bash -c '
    pattern=$(tr "\n" "\x1e" < ./compromised_header.txt)
    for file; do 
        tr "\n" "\x1e" < "$file" | grep -qF "$pattern" && echo "$file"
    done
' bash
3
  • 1
    this will match any line from compromised_header.txt instead of matching entire content of compromised_header.txt as single search
    – Sundeep
    Apr 5, 2021 at 14:12
  • @Sundeep: I believe the idea of -z is to see the input file as one line and the searched files as well? so it would conserve the order of lines (and enforce consecutive lines matching) Apr 5, 2021 at 15:07
  • 1
    @OlivierDulac I checked an example and it matched single line from compromised_header.txt.. pattern isn't affected by -z flag, only the definition of 'line'
    – Sundeep
    Apr 5, 2021 at 15:13
2

Given the input:

$ head -n -0 ?-compromised-file.txt
==> a-compromised-file.txt <==
this content is in a-compromised-file.txt
some unique string 1
some other unique string 2
another unique string 3
unrelated line x
unrelated line y
unrelated line z

==> b-compromised-file.txt <==
this content is in b-compromised-file.txt
some unique string 1
another unique string 3
some other unique string 2
unrelated line x
unrelated line y
unrelated line z

==> c-compromised-file.txt <==
this content is in c-compromised-file.txt
some unique string 1
some unique string 1
some other unique string 2
another unique string 3
unrelated line x
unrelated line y
unrelated line z

... this Perl script:

while (<>) {
    # Read the pattern from the first file.
    $pattern .= $_;
    last if eof;
}
# Search remaining files for the pattern.
while (<>) {
    # If existing buffer continues matching pattern, or if
    # the current line matches the beginning of pattern...
    if (($buf .= $_) eq substr($pattern,0,length($buf))
        or (($buf = $_) eq substr($pattern,0,length($buf)))
    ) { 
        # If we successfully match the whole pattern, move along.
        if ($pattern eq $buf) {
            print $ARGV, "\n";
            $buf = q{};
            do {$_ = <>} until eof; # skip to end of current file
        }
    }
    else { $buf = q{}; }
}

... produces these results:

$ find . -name '*-compromised-file.txt' | xargs perl above.pl compromised_header.txt
./a-compromised-file.txt
./c-compromised-file.txt

It doesn't gobble memory, isn't subject to regex interpolation, and stops looking for additional matches in a file if one is already found. I hope it's commented well enough to follow.

Concerning some of the other answers/comments, be careful with slurp mode if you have large files, as you may hit memory constraints.

Note that the c-compromised-file.txt shown above fails at least one of the other answers. (I haven't enough reputation yet to comment there.)

0
1

We can accomplish the detection of the header using GNU sed and supplying multiple files but using the -s option to keep the streams separated.

ref=$(< compromised_header.txt sed -e 's:[\&/]:\\&:g;$!s:$:\\:')
find dir -type f -exec \
sed -sEn "
  1{x;/^\$/s/.*/$ref/;x;}"'
  /\n/!G
  /^([^\n]*)\n\1(\n|$)/!{
    G;/^[^\n]*\n(.*)\n\1$/d
    s/\n.*//;s/^/\n/;D
  }
  s///;/^$/{F;:n;n;$!bn;}
  $d;N
  s/(.*)((\n).*)/\2\3\1/;D
' {} +

Store the compromised header after duly making it pluggable in sed's RHS . Then we compare it with the incoming data. For a match, we clip the top element of hold and read the next line. This stops when the hold is emptied and filename printed. And should there be a nomatch, we restore the hold portion in pattern space that may have been consumed till that point and redo activities from that point.

#==============================

Below is another method that involves various linux utilities. The general idea isvthat the first header line number in a file are determined via grep. Next starting from this a chunk equal in size to the header file is extracted via the ed editor and compared with header. Print filename on a match. The looping is done via find.

find . -type f -exec sh -c '
  tick='\\\''
  ref=$1;shift
  L1=$(< "$ref" head -n1)
  len=$(dc -e "$(< "$ref" wc -l) 1-f")
  for f do
  set -- $(grep -nxFe "$L1" < "$f" | cut -d: -f1)
  for lnum do
    ed -s "$f" <<eof |\
    cmp -s - "$ref" && {\
    printf "%s\\n" "$f";break;}
${lnum}kx
${tick}x,${tick}x+${len}p
Q
eof
  done;done
' find-sh compromised_header.txt {} +

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .