Find files that are named from a list of patterns in file #1 and grep inside them for patterns found in file #2

Question

I am trying to fix a bash script that is supposed to:

1. Go to a directory (RESULT_DIR)
2. Find csv files that have matched names from a list file (Names.txt)
3. Grep from patterns listed in (Patterns.txt) found in files from step 2
4. Take that output and deposit it into a MatchingResults.txt

So far, I am getting the "argument list too long" error. But I am not sure how to fix this. I would appreciate some help on how to fix it, since I may have made mistakes I am not seeing.

#! /usr/bin/env bash

RESULT_DIR="$HOME/.../Results/"

NAMES="$HOME/.../Names.txt"

PATTERNS="$HOME/.../Patterns.txt"

cd "$RESULT_DIR" && grep -f "$PATTERNS" $(find $(cat "$NAMES").csv) >> MatchingResults.txt

Update: Here is what NAMES and PATTERNS look like so you can get a better idea of what I mean. Sorry this was lacking!

"NAMES"
"O60333"
"P52209"
"Q8N2Z9"
"O00230"
"O00273"
"O00468"
"O75381"
"Q86V15"
"E7ERA6"
"Q96HA4"
"K7EPZ7"
"H3BM07"
"H0YBK5"
"G8JLG8"
"Q13148"
"O00187"

and

"PATTERNS"
"R381P"
"T95A"
"E112K"
"R136G"
"R140Q"
"S149L"
"R173Q"
"S184A"
"E193G"
"V260M"
"P291L"
"H313Y"
"P328L"

Files in RESULT_DIR would have names such as "A12345.csv".

I would expect cat "$NAMES" would output the contents of Names.txt as above so that they could be read line by line as arguments for grep -f patterns filestolookthrough.

What I'm trying to do with $(find $(cat "$NAMES").csv) is go through the names to match them in the results directory line by line to then match the grep -f pattern solely in that particular matching file.

My error output only says line 8: /usr/bin/find: Argument list too long

Answer 1

You should ideally add some more Details & Examples, Eg ::
(A) Contents of RESULT_DIR & NAMES & PATTERNS ;
(B) What you think cat "$NAMES" will output ;
(C) What you think $(find $(cat "$NAMES").csv) will output [[ IMPORTANT :: This will not add the extension .csv to all names in the file $NAMES !! Only the last entry in file $NAMES will get the extension .csv !! ]] ;
(D) What is the exact error output you are getting ; ....

Without these Details, we have to guess on the Problem and give fixes. With these Details, we might be able to actually come up with a Solution.

Pending your updates to the Question, here are 3 guesses on what is going on:
(1) It is likely that NAMES contains too many lines and find is getting a very long list of arguments.
(2) It is also likely that NAMES contains few lines and find is generating a very long list of files and this goes to grep which gets a very long list of arguments.
(3) PATTERNS has too many lines, but this is unlikely.

(4) It may likely turn out to be entirely something else, but you have to update your Question with Details.

Based on OP updates, here is a Solution:

I think, NAMES file should already be having the .csv extension; This is simple to edit via vim or via whichever script generated the list originally. This way, it is not necessary to wrongly use cat and add .csv to only the last entry.
More-over, the list of files do not require quotes.
Next, PATTERNS file should not have quotes, which will not match the required text.

With these changes, the given script will contain this last line, where .csv is eliminated:
cd "$RESULT_DIR" && grep -f "$PATTERNS" $(find $(cat "$NAMES")) >> MatchingResults.txt

Now, if the NAMES are always inside RESULT_DIR (or a known directory) then that PATH can be included in NAMES, line-by-line; and we can eliminate find too.
With that change, the last line of the given script will be this:
cd "$RESULT_DIR" && grep -f "$PATTERNS" $(cat "$NAMES") >> MatchingResults.txt

Suggest OP try this with a small list in PATTERNS & NAMES. With that working, we can move on to using xargs with too many arguments in either PATTERNS or NAMES.
Suggest OP to also list the approximate number of lines in PATTERNS & NAMES, and exact error output, when executing the script.

UPDATE: The long list from cat is going to find which is throwing the error. Here is a way around that:

  
cd "$RESULT_DIR"  
for fn in $(cat "$NAMES") #### "Iterate over all files listed in $NAMES, Ensure that there are no quotes inside $NAMES file"  
do  
    grep -f "$PATTERNS" ${fn}.csv  
    #### "Not necessary to add .csv if $NAMES file already has it inside line-by-line, else add it here"  
    #grep -f "$PATTERNS" $(find ${fn}.csv)  
    #### "OR use this alternative, in case the list of files in $NAMES is not immediately inside $RESULT_DIR, but is in some subdirectory"  
done >> MatchingResults.txt  

Now, there is no find and there is no way to get the error from find about arguments !

Answer 2

I think this does what you want. I assume the $NAMES file are filenames that need the .csv extension added then each one is grepped for the patterns.

cd "$RESULT_DIR" && sed 's/$/.csv/' "$NAMES" | xargs grep -Hf "$PATTERNS" >> MatchingResults.txt