Check whether all lines of file occur in different file

Question

I got two files: file1 with about 10 000 lines and file2 with a few hundred lines. I want to check whether all lines of file2 occur in file1. That is: ∀ line ℓ ∈ file2 : ℓ ∈ file1

Should anyone not know what these symbols mean or what "check whether all lines of file2 occur in file1" means: Several equivalent lines in either files don't influence whether the check returns that the files meet the requirement or don't.

How do I do this?

Answer 1

comm -13 <(sort -u file_1) <(sort -u file_2)

This command will output lines unique to file_2. So, if output is empty, then all file_2 lines are contained in the file_1.

From comm's man:

   With  no  options,  produce  three-column  output.  Column one contains
   lines unique to FILE1, column two contains lines unique to  FILE2,  and
   column three contains lines common to both files.

   -1     suppress column 1 (lines unique to FILE1)

   -2     suppress column 2 (lines unique to FILE2)

   -3     suppress column 3 (lines that appear in both files)

Answer 2

[ $(grep -cxFf file2 <(sort -u file1)) = $(sort -u file2 | wc -l) ] && 
  echo all there || 
  echo some missing

If the number of matches from file2 in (the unique lines of) file1 is the same as the number of unique lines in file2, then they're all there; otherwise, they aren't.

Answer 3

Using GNU awk where it does support specific length(array) feature (and some other awk implementation which may can support) and not required if files are sorted.

gawk 'FNR==NR{seen[$0];next} ($0 in seen){delete seen[$0]};
    END{print (!length(seen))?"Matched":"Not Matched"}' file2 file1

This is reading file2 into an array called seen with the key as entire line of file2.

Then read file1 and for each line if matched with lines in array seen then delete that key.

At the end if the array was empty means all lines in file2 exist in file1 and will print Matched, otherwise will display Not Matched.

---

For the compatibility in all awk implementations.

awk 'FNR==NR{seen[$0];next} ($0 in seen){delete seen[$0]};
    END{for(x in seen);print (!x)?"Matched":"Not Matched"}' file2 file1

To ignoring empty lines/or lines with whitespaces only if in file2, you would need to add NF to the condition in NR==FNR && NF {... to skip reading them into the array.

Answer 4

Using comm you can find lines that are common in both files.

comm -12 file1 file2

Have a look at man comm for more details

Answer 5

diff -q <(sort -u file2) <(grep -Fxf file2 file1 | sort -u)

will produce no output if file1 contains all the lines in file2 and exit with status 0, otherwise it will print something like

Files /proc/self/fd/11 and /proc/self/fd/12 differ

and exit with status 1

Answer 6

Use a Python program:

#!/usr/bin/env python3
import sys

def open_arg(path):
    return sys.stdin if path == '-' else open(path)

def strip_linebreak(s):
    return s[:-1] if s.endswith('\n') else s

with open_arg(sys.argv[1]) as pattern_file:
    patterns = set(map(strip_linebreak, pattern_file))

with open_arg(sys.argv[2]) as dataset_file:
    for l in map(strip_linebreak, dataset_file):
        patterns.remove(l)
        if not patterns:
            break

sys.exit(int(bool(patterns)))

Usage:

python3 contains-all.py file2 file1

The program exit status indicates whether all patterns of file 2 were matched:

• 0 (success) means all patterns were matched.
• 1 (failure) means some patterns weren't matched.

To query the exit status in a shell (script) you can either use the $? special variable or other expressions that evaluate command exit status, e. g. short circuit operators && and || and conditional expressions like if or while. Example:

if python3 compare-all.py file2 file1 && some-other --condition; then
    # do stuff
fi

Answer 7

combine from moreutils will show you all the lines in file2 that aren't in file1 with:

combine file2 not file1

Then you can count the number of lines by piping it to wc -l, like:

if [ $(combine file2 not file1 | wc -l) != 0 ]; then
  echo "lines missing"
else
  echo "You're fine"
fi