Linux search for a string only in a given set of files

Question

There are several files in a directory. I am trying to search and find all such strings ending with a given string. Instead of searching in all the files present in the directory, I would like to search only in a given set of file names. Finally the output should be each filename along with the string occurrences found in that file with a semicolon separator.

The simplified test case is: there are 5 files in the directory:

file.a.txt
file.b.txt
file.c.txt
file.d.txt
file.e.txt

There is also a file named searchFiles.txt that contains the first 3 filenames from above list. So I want to search for the string only in the filenames listed in searchFiles.txt.

I have tried:

for i in $(cat searchFiles.txt); do grep -o '[^ ]*_XYZ' /dev/null $i ; done | awk -F: '{a[$1]=a[$1]";"$2;} END{for (x in a) print x ":" substr(a[x],2);}'

but the output says

: No such file or directory
: No such file or directory
file.c.txt:FOUND1_XYZ;FOUND2_XYZ

So somehow it is able to search only in the last file name given in searchFiles.txt but the other initial files couldn't be found and thus throws the error 'No such file or directory'

The output I am expecting is:

file.a.txt:FOUNDSTR_XYZ
file.b.txt:FOUNDSTR1_XYZ;FOUNDSTR2_XYZ;FOUNDSTR3_XYZ
file.c.txt:FOUND1_XYZ;FOUND2_XYZ

I was also trying to find if the "-name" flag of "find" command could be helpful here, but couldn't quite get it on how exactly to feed the list of files from searchFiles.txt here. The below attempt went in vain.

find . -type f -name `cat searchFiles.txt` -exec grep -o '[^ ]*_XYZ' /dev/null {} \;

Also:

• There can be upto few 1000s of files in a directory and the search file names in searchFiles.txt can be few 100s of file names.

• The file names can be anything and doesn't follow any pattern as such.

• The file names provided in searchFiles.txt can be partial names for instance a.txt instead of file.a.txt which means the initial static portion of the file name "file." may or may not be present in searchFiles.txt.

• preferably looking for a one-liner commands instead of a shell script

Any help on this please?

Answer 1

You should be able to the whole thing with GNU awk, like:

find . -type f -print0 |
  gawk '
    step == 1 {files[$0]; next} # record file names in "files" array
    step == 2 {
      # determine which files to look into (added to ARGV array for
      # processing in step 3)
      if ($NF in files) ARGV[ARGC++] = $0; next
    }
    NF {
      # record all matches (here in fields matched by FPAT)
      $1 = $1 # force a rebuild of $0 joining fields with OFS
      matches[FILENAME] = matches[FILENAME] \
                          (matches[FILENAME] == "" ? "" : OFS) \
                          $0
    }
    END {
      for (file in matches)
        print file ": " matches[file]
    }' step=1 searchFiles.txt \
       step=2 RS='\0' FS=/ - \
       step=3 RS='\n' FPAT='[^ ]*_XYZ' OFS=';'

Above, the filenames are matched exactly against the files stored in searchFiles.txt. If the lines of that file are meant to be a list of suffixes, you could build a regexp instead of an associative array:

find . -type f -print0 |
  gawk '
    step == 1 {
      gsub(/[][^$*()+{}?\\.|]/, "\\\\&") # escape regexp operators
      regex = regex sep $0; sep = "|"
      next
    }
    step == 2 {
      # determine which files to look into (added to ARGV array for
      # processing in step 3)
      if ($NF ~ ("(" regex ")$")) ARGV[ARGC++] = $0; next
    }
    NF {
      # record all matches (here in fields matched by FPAT)
      $1 = $1 # force a rebuild of $0 joining fields with OFS
      matches[FILENAME] = matches[FILENAME] \
                          (matches[FILENAME] == "" ? "" : OFS) \
                          $0
    }
    END {
      for (file in matches)
        print file ": " matches[file]
    }' step=1 searchFiles.txt \
       step=2 RS='\0' FS=/ - \
       step=3 RS='\n' FPAT='[^ ]*_XYZ' OFS=';'

If you need to obfuscate it, you can put it on one line:

find . -type f -print0|gawk '!s{gsub(/[][^$*()+{}?\\.|]/,"\\\\&");r=r p $0;p="|";next};s==2{if($NF~("("r")$"))ARGV[ARGC++]=$0;next};NF{$1=$1;m[FILENAME]=m[FILENAME](m[FILENAME]==""?"":OFS)$0};END{for(f in m)print f":"m[f]}' searchFiles.txt s=2 RS=\\0 FS=/ - s=3 RS=\\n FPAT='[^ ]*_XYZ' OFS=\;

Those make no assumption about what characters the file names and contents may contain other than they have to be valid characters in the locale. Suffixes can't have newline character, but that's a constraint driven by the format of your searchFiles.txt.

Answer 2

I'm going to assume you have fixed the DOS-style line endings as discussed in comments, and that searchFiles.txt does not really contain empty lines.

The -name test of find takes exactly one filename pattern. Patterns may contain shell glob characters, but these should be protected from premature filename generation by the shell. You can concatenate multiple such tests with logical OR using -o but you need to be careful about operator precedence.

If your shell supports arrays, one way you might be able to do that is as follows (I'm using bash here but similar procedures should be possible in other shells):

files=( -false )
while IFS= read -r f || [ -n "$f" ]; do files+=( -o -name "*$f"); done < searchFiles.txt

This should result in ${files[@]} expanding to the alternation

-false -o -name *file.a.txt -o -name *file.b.txt -o -name *file.c.txt -o -name *file.d.txt -o -name *file.e.txt

which you can then use in your find command like

find . \( "${files[@]}" \) -exec grep -Ho '[^ ]*_XYZ' {} +

(I omitted the dummy file /dev/null in favour of adding the -H option). If the number of files in searchFiles.txt is too large, this approach may fail due to ARG_MAX limits. You could possibly work around such a limit by splitting searchFiles.txt into multiple smaller files.

Answer 3

You can filter the filenames into your directory with a grep -f through the text file with the names to include (allowing partial match). Then do a massive grep into these files for the search pattern and finally format the result with a small awk.

Using GNU bash:

grep -Ff filenames.txt <(printf '%s\n' *) |
    xargs -d '\n' grep -oH '[^[:space:]]*_XYZ$' | awk -F: '
        {f[$1] = f[$1] ? f[$1] ";" $2 : $0}
        END {for (x in f) print f[x]}'

Some assumptions (there are not all clear into the question):

• Your filenames are convenient, no newlines, no colons (for the grep output). Spaces are handled.
• There are no subdirectories matching into there or else the second grep would throw a message but it will return results.
• The second grep looks for the pattern at the end of the line. You can modify it if you want to match end of words.
• -H for grep is used for the corner case of one file, to print filenames into the output (this is the default when there are two or more files)