How do I compare all files in dir to another file and print output of common lines

Question

First I have a dir/ consisting of 100 .txt files

john_1.txt, john_2.txt, john_3.txt, john_4.txt etc, that consists of ex. id;email;name;phone and then I have one other file emails.txt that consists of e-mail adresses listed line by line.

How do I compare all the files to this emails file and print the common lines.

grep or diff?

Answer 1

With awk, something like this:

awk -F\; 'NR == FNR {A[$0]=1; next}; A[$2] == 1;' emails.txt john*.txt

NR == FNR tests if the record (line) number counted over all files is the the same as the record number in current file, which is a funny way of testing if this is the first file. If it is, we take the full line read ($0) and use it as a key to an associative array called A, setting the value to one, and jumping to the next line of input. If it wasn't the first file (the jump wasn't taken), take the second field ($2) on the line, separated by semicolons (set by -F) and see if the corresponding value in the array A is one. If it is, the default action is to print the whole line. You could add, say, {print $1} to only print the first field.

---

With grep you could just do

grep -F -f emails.txt john*.txt

(-F for fixed string patterns, -f to read patterns from a file.) But this will print partial matches too, and also matches from other fields in the file, but that might not be a problem with names, phone numbers and email addresses.

Though that could be worked around by adding semicolons to the start and end of each line used as a pattern. This should work if process substitution is supported:

grep -F -f <(sed -e 's/^/;/' -e 's/$/;/' emails.txt) john*.txt