4

FILE-A has 100,000 lines. FILE-B is 50 search terms. I'm looking to complete the search of FILE-A (CSV or TXT) with the various terms from FILE-B (CSV or TXT) AND -- here is the kicker -- save the results in individual TXT files based off the search terms from FILE-B.

Example:

FILE-A

123
45678
1239870
2349878
39742366876
41967849
789
910
2378
6723

FILE-B

1
2
23
78

Results = "1.txt" with all matching lines from FILE-A, "2.txt" with all lines matching from FILE-A, "23.txt", "78.txt" and so on. So if FILE-B has 50 search terms, I would end up with 50 TXT files, named with the search term, assuming at least one hit with said term from FILE-A.

I have searched using "fgrep -f FILE-B.txt FILE-A.csv >> output.txt" This puts all of the search terms from FILE-B found in FILE-A into one "output.txt". I'm instead looking to separate them into individual text files.

0

2 Answers 2

5

Grep + Xargs

xargs -d '\n' sh -c '
    for term; do grep "$term" fileA > "$term.txt"; done
' xargs-sh < fileB

Improved by cas.

Grep + Shell

Generally using shell loops to read a file is bad practice, but here fileB is much smaller than fileA so it won't significantly hurt performance.

while IFS= read -r term; do
    grep "$term" fileA > "$term.txt"
done < fileB

Awk

awk 'NR==FNR{pat[$0];next}{for(term in pat){if($0~term){print>term}}}' fileB fileA
  • NR==FNR{pat[$0];next} reads the first file given as an argument and puts each line in the array pat.
  • {for(term in pat){if($0~term){print>term}}} is self-explainable: For each term in the array, test if the current line matches the term and print it to a file named accordingly if yes.

Not all Awks will allow for many files to be open at the same time. One way to tackle this, as suggested by Ed Morton, is to use a close statement and to use the append operator:

awk 'NR==FNR{pat[$0];next}{for(term in pat){if($0~term){print>>term;close(term)}}}' fileB fileA
1
  • For the awk script, instead of closing and opening an output file for each write (there is significant overhead in doing that), I'd use another variable, say prev_term, and do something like if (term != prev_term) {close(prev_term); prev_term=term}. This will only close the output file when term changes, not every time output is written.
    – cas
    Aug 9, 2021 at 16:56
2

This should be efficient as it uses a first pass with grep -F (which is very fast) to output just the lines from FILE-A that match lines from FILE-B (which presumably will be a much smaller number of lines than the original 100,000 in FILE-A) so then the awk script doesn't have so many lines from FILE-A to compare and can loop through those when reading FILE-B instead of looping through the FILE-B lines when reading FILE-A and so is able to only open/close 1 output file for each line of FILE-B rather than open/close 1 output file at a time for every line of FILE-B * every line that matches it in FILE-A to avoid the potential "too many open files" error.

$ cat tst.sh
#!/usr/bin/env bash

grep -F -f 'FILE-B' 'FILE-A' |
awk '
    NR==FNR{ lines[++numLines]=$0; next }
    {
        close(out)
        out = $0 ".txt"
        for (i=1; i<=numLines; i++) {
            line = lines[i]
            if (index(line,$0)) {
                print line > out
            }
        }
    }
' - 'FILE-B'

$ ./tst.sh

$ head -100 *.txt
==> 1.txt <==
123
1239870
41967849
910

==> 2.txt <==
123
1239870
2349878
39742366876
2378
6723

==> 23.txt <==
123
1239870
2349878
39742366876
2378
6723

==> 78.txt <==
45678
2349878
41967849
789
2378

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .