0

I am using grep to find matching lines between two files.

grep -w -f file1.txt file2.txt > file3.txt

I know there is the -v flag that can give me lines from file2.txt that don't match lines in file1.txt. Is there a way to get the lines from file1.txt that have no match in file2.txt? Switching their places won't work since file2.txt has more text in each line.

Essentially, file1.txt is a text file containing a single column of IDs without any whitespaces, while file2.txt is a table, in which the columns are separated with tabs and one column has IDs like in file1.txt. So what I am trying to find are the lines of file2.txt that do not have a matching ID in file1.txt.

Actually, now I think I have found an answer in another question: "Ungrep" - which patterns aren't matched

6
  • 1
    Sounds like you should consider using the diff command
    – fpmurphy
    Sep 2, 2020 at 19:45
  • 1
    I would recommend comm
    – pLumo
    Sep 2, 2020 at 20:09
  • Thanks for suggestions, but it was my understanding that diff and comm both require the two files be similarly formatted and lines containing the same things. As I said the files are different in that file2 has strings from file1 but the all lines have more text than just those strings.
    – danvoronov
    Sep 2, 2020 at 20:40
  • 1
    Some examples of the two files might help, especially as to whether the shorter lines are initial parts of the others, or may be in the middle, and whether this is free text or delimited fields. Also approximate data volumes. Also, whether one file1 row may match multiple file2 rows. I suspect awk would be the most capable method, but a full (n x m) match may become pathologically slow on large files. Sep 2, 2020 at 21:06
  • I note the -w flag on your grep, which requires whole words (defined as letters, digits and underscore). Is it correct to assume neither file should be interpreted as a pattern, amount of whitespace is not significant, punctuation should be ignored, etc? Sep 2, 2020 at 23:43

2 Answers 2

0

This answer is a combination of @danvoronov's initial post, and @waltinator's solution to getting the unused keys, with improvements to both.

I have about 60 lines of Bash and awk which construct test files using random selections from English language sets in the Princeton CS50 course material. I also have 30 lines of Bash and awk which solve this problem in a single process. This needs performance improvement so I will not as yet post it.

My script and the one posted below produce identical results. For reference and checking purposes, each output text is preceded by the line number in the original file. The test files consist of 15000 shorter records in file1 (12000 that match file2 and 3000 that do not, average length 92 chars), and 20000 full lines in file2 (average length 129 chars).

The keys are taken from a random selection of the larger records, taking a random number of words (but at least four) starting from a random word in the line.

Note that a single key can match multiple records. We do not attempt to enforce one-on-one pairing.

The posted script below runs on that data volume in about 1m:30s (my awk in 5m:45s).

The data from file1 is not likely to be valid regular expressions, so the -F option has to be used in grep. If plain text is treated as an RE, many characters like * + . will match unexpectedly, and others like ( ) [ ] | will throw errors. -F can also be much faster (I found a 600x improvement -- I want to know what that algorithm is!).

The grep for unused keys just uses a -q option, and the status shows whether a match occurred. This saves two processes (a subshell and a wc -l for each line in file1), and on average it also halves the data read, because it exits on first match rather than reading the whole file.

#! /bin/bash

    grep -n -w -F -f file1.txt file2.txt > file3.txt

    while read -r Key; do
        (( ++Fnr ))
        grep -q -w -F -- "$Key" file2.txt || printf '%d:%s\n' "${Fnr}" "$Key"
    done < file1.txt > file4.txt

As some of the initial questions are unanswered (whether the data is in fields, whether whitespace matters, any data volumes), comments are invited.

0

Process the lines of file1.txt individually:

for str in $(cat file1.txt) ; do
    num=$(grep -w "$str" file2.txt | wc -l )
    if [[ $num -eq 0 ]] ; then
        echo "$str"
    fi
done
1
  • Some issues in there: (a) The $(cat ..) construct returns individual words, not lines. (b) Without sample data, it is likely file1 contains some RE special characters, so we need grep -F. (c) We don't know file volumes, so running a grep per line of file1 may not be sensible. Sep 3, 2020 at 17:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.