How to grep non-matching lines from first file when grepping files?

Question

I am using grep to find matching lines between two files.

grep -w -f file1.txt file2.txt > file3.txt

I know there is the -v flag that can give me lines from file2.txt that don't match lines in file1.txt. Is there a way to get the lines from file1.txt that have no match in file2.txt? Switching their places won't work since file2.txt has more text in each line.

Essentially, file1.txt is a text file containing a single column of IDs without any whitespaces, while file2.txt is a table, in which the columns are separated with tabs and one column has IDs like in file1.txt. So what I am trying to find are the lines of file2.txt that do not have a matching ID in file1.txt.

Actually, now I think I have found an answer in another question: "Ungrep" - which patterns aren't matched

Answer 1

This answer is a combination of @danvoronov's initial post, and @waltinator's solution to getting the unused keys, with improvements to both.

I have about 60 lines of Bash and awk which construct test files using random selections from English language sets in the Princeton CS50 course material. I also have 30 lines of Bash and awk which solve this problem in a single process. This needs performance improvement so I will not as yet post it.

My script and the one posted below produce identical results. For reference and checking purposes, each output text is preceded by the line number in the original file. The test files consist of 15000 shorter records in file1 (12000 that match file2 and 3000 that do not, average length 92 chars), and 20000 full lines in file2 (average length 129 chars).

The keys are taken from a random selection of the larger records, taking a random number of words (but at least four) starting from a random word in the line.

Note that a single key can match multiple records. We do not attempt to enforce one-on-one pairing.

The posted script below runs on that data volume in about 1m:30s (my awk in 5m:45s).

The data from file1 is not likely to be valid regular expressions, so the -F option has to be used in grep. If plain text is treated as an RE, many characters like * + . will match unexpectedly, and others like ( ) [ ] | will throw errors. -F can also be much faster (I found a 600x improvement -- I want to know what that algorithm is!).

The grep for unused keys just uses a -q option, and the status shows whether a match occurred. This saves two processes (a subshell and a wc -l for each line in file1), and on average it also halves the data read, because it exits on first match rather than reading the whole file.

#! /bin/bash

    grep -n -w -F -f file1.txt file2.txt > file3.txt

    while read -r Key; do
        (( ++Fnr ))
        grep -q -w -F -- "$Key" file2.txt || printf '%d:%s\n' "${Fnr}" "$Key"
    done < file1.txt > file4.txt

As some of the initial questions are unanswered (whether the data is in fields, whether whitespace matters, any data volumes), comments are invited.

Answer 2

Process the lines of file1.txt individually:

for str in $(cat file1.txt) ; do
    num=$(grep -w "$str" file2.txt | wc -l )
    if [[ $num -eq 0 ]] ; then
        echo "$str"
    fi
done