Search File A with Terms from File B and save output to individual TXT files based off search term from File B

Question

FILE-A has 100,000 lines. FILE-B is 50 search terms. I'm looking to complete the search of FILE-A (CSV or TXT) with the various terms from FILE-B (CSV or TXT) AND -- here is the kicker -- save the results in individual TXT files based off the search terms from FILE-B.

Example:

FILE-A

123
45678
1239870
2349878
39742366876
41967849
789
910
2378
6723

FILE-B

1
2
23
78

Results = "1.txt" with all matching lines from FILE-A, "2.txt" with all lines matching from FILE-A, "23.txt", "78.txt" and so on. So if FILE-B has 50 search terms, I would end up with 50 TXT files, named with the search term, assuming at least one hit with said term from FILE-A.

I have searched using "fgrep -f FILE-B.txt FILE-A.csv >> output.txt" This puts all of the search terms from FILE-B found in FILE-A into one "output.txt". I'm instead looking to separate them into individual text files.

Answer 1

Grep + Xargs

xargs -d '\n' sh -c '
    for term; do grep "$term" fileA > "$term.txt"; done
' xargs-sh < fileB

^{Improved by cas.}

Grep + Shell

Generally using shell loops to read a file is bad practice, but here fileB is much smaller than fileA so it won't significantly hurt performance.

while IFS= read -r term; do
    grep "$term" fileA > "$term.txt"
done < fileB

Awk

awk 'NR==FNR{pat[$0];next}{for(term in pat){if($0~term){print>term}}}' fileB fileA

• NR==FNR{pat[$0];next} reads the first file given as an argument and puts each line in the array pat.
• {for(term in pat){if($0~term){print>term}}} is self-explainable: For each term in the array, test if the current line matches the term and print it to a file named accordingly if yes.

Not all Awks will allow for many files to be open at the same time. One way to tackle this, as suggested by Ed Morton, is to use a close statement and to use the append operator:

awk 'NR==FNR{pat[$0];next}{for(term in pat){if($0~term){print>>term;close(term)}}}' fileB fileA

Answer 2

This should be efficient as it uses a first pass with grep -F (which is very fast) to output just the lines from FILE-A that match lines from FILE-B (which presumably will be a much smaller number of lines than the original 100,000 in FILE-A) so then the awk script doesn't have so many lines from FILE-A to compare and can loop through those when reading FILE-B instead of looping through the FILE-B lines when reading FILE-A and so is able to only open/close 1 output file for each line of FILE-B rather than open/close 1 output file at a time for every line of FILE-B * every line that matches it in FILE-A to avoid the potential "too many open files" error.

$ cat tst.sh
#!/usr/bin/env bash

grep -F -f 'FILE-B' 'FILE-A' |
awk '
    NR==FNR{ lines[++numLines]=$0; next }
    {
        close(out)
        out = $0 ".txt"
        for (i=1; i<=numLines; i++) {
            line = lines[i]
            if (index(line,$0)) {
                print line > out
            }
        }
    }
' - 'FILE-B'

$ ./tst.sh

$ head -100 *.txt
==> 1.txt <==
123
1239870
41967849
910

==> 2.txt <==
123
1239870
2349878
39742366876
2378
6723

==> 23.txt <==
123
1239870
2349878
39742366876
2378
6723

==> 78.txt <==
45678
2349878
41967849
789
2378