I got two files: file1 with about 10 000 lines and file2 with a few hundred lines. I want to check whether all lines of file2 occur in file1. That is: ∀ line ℓ ∈ file2 : ℓ ∈ file1

Should anyone not know what these symbols mean or what "check whether all lines of file2 occur in file1" means: Several equivalent lines in either files don't influence whether the check returns that the files meet the requirement or don't.

How do I do this?

  • 2
    May those files have duplicated lines? If file2 contains 2 lines A, do you need file1 to contain at least 2 lines A? Commented Oct 12, 2017 at 19:57
  • 2
    @StéphaneChazelas All lines (in both files) are guaranteed to be unique.
    – UTF-8
    Commented Oct 12, 2017 at 21:31
  • 1
    @UTF-8 That would be an important detail to edit into your question.
    – David Z
    Commented Oct 13, 2017 at 1:45
  • 2
    @DavidZ Not anymore as the existing answers don't rely on that guarantee. So by editing the question now, I'd reduce the apparent scope of the answers.
    – UTF-8
    Commented Oct 13, 2017 at 10:08
  • @UTF-8 I suppose so, although the question is a little ambiguous without it, e.g. if a given line occurs 5 times in file2, does that line also have to occur 5 times in file1 (as opposed to only once)? If you had that requirement, it doesn't look like any of the existing answers would work, so I'd suggest at least editing in something that makes it clear that that's not what you mean.
    – David Z
    Commented Oct 13, 2017 at 18:06

7 Answers 7

comm -13 <(sort -u file_1) <(sort -u file_2)

This command will output lines unique to file_2. So, if output is empty, then all file_2 lines are contained in the file_1.

From comm's man:

   With  no  options,  produce  three-column  output.  Column one contains
   lines unique to FILE1, column two contains lines unique to  FILE2,  and
   column three contains lines common to both files.

   -1     suppress column 1 (lines unique to FILE1)

   -2     suppress column 2 (lines unique to FILE2)

   -3     suppress column 3 (lines that appear in both files)
  • @don_crissti True. Fixed: the -u option added to the sort command. Now, only unique lines are left in both sorted files.
    – MiniMax
    Commented Oct 12, 2017 at 19:58
  • Awesomely simply solution! Is this syntax applicable to any program which expects files? I always thought the < piped into stdin. Does the bracket term change this?
    – UTF-8
    Commented Oct 12, 2017 at 21:43
  • 2
    @UTF-8 It is called Process substitution. You can read here about it. And yes, it behaves like a temporary file, so it can be used instead of real files in any programs, which expects files.
    – MiniMax
    Commented Oct 12, 2017 at 21:57
  • If this is something you do often, you might want to store file_1 in presorted form. Saves both typing and time. Commented Oct 13, 2017 at 8:47
  • 7
    @minimax Good comment except for the "any". Process substitution, while wonderful, cannot be used in all cases, because the resulting "files" are streams and not real files. This means that they are not "seekable" like a normal file is, and can only be used when the program reads the file normally from the beginning, and not when the program uses some file-only functionality such as seeking to a specific point or rewinding to start over from the beginning. Happily, most programs simply read() their files, and so process substitution works with most programs, but not "any" programs.
    – Law29
    Commented Oct 13, 2017 at 8:50
[ $(grep -cxFf file2 <(sort -u file1)) = $(sort -u file2 | wc -l) ] && 
  echo all there || 
  echo some missing

If the number of matches from file2 in (the unique lines of) file1 is the same as the number of unique lines in file2, then they're all there; otherwise, they aren't.


Using GNU awk where it does support specific length(array) feature (and some other awk implementation which may can support) and not required if files are sorted.

gawk 'FNR==NR{seen[$0];next} ($0 in seen){delete seen[$0]};
    END{print (!length(seen))?"Matched":"Not Matched"}' file2 file1

This is reading file2 into an array called seen with the key as entire line of file2.

Then read file1 and for each line if matched with lines in array seen then delete that key.

At the end if the array was empty means all lines in file2 exist in file1 and will print Matched, otherwise will display Not Matched.

For the compatibility in all awk implementations.

awk 'FNR==NR{seen[$0];next} ($0 in seen){delete seen[$0]};
    END{for(x in seen);print (!x)?"Matched":"Not Matched"}' file2 file1

To ignoring empty lines/or lines with whitespaces only if in file2, you would need to add NF to the condition in NR==FNR && NF {... to skip reading them into the array.

  • length(array) is gawk-only AFAIK; it definitely isn't POSIX. Commented Oct 13, 2017 at 4:15
  • @dave_thompson_085 Correct, I have updated my answer. thanks Commented Oct 13, 2017 at 5:09

Using comm you can find lines that are common in both files.

comm -12 file1 file2

Have a look at man comm for more details

  • Correct it's returning common lines in both files, but this doesn't provide an answer to the OP's Q where if you had a line in file2 which doesn't exit in file1, so all lines of file2 doesn't exist in file1. Commented Oct 12, 2017 at 18:12
  • 1
    files should be sorted. From man "comm - compare two sorted files line by line".
    – MiniMax
    Commented Oct 12, 2017 at 19:50
  • @MiniMax is right. This doesn't work. The other answer utilizing comm contains a solution which isn't obviously incorrect. When I run your command, I get warnings that the files aren't in sorted order and lots of lines which definitely are in both files.
    – UTF-8
    Commented Oct 12, 2017 at 21:41
diff -q <(sort -u file2) <(grep -Fxf file2 file1 | sort -u)

will produce no output if file1 contains all the lines in file2 and exit with status 0, otherwise it will print something like

Files /proc/self/fd/11 and /proc/self/fd/12 differ

and exit with status 1


Use a Python program:

#!/usr/bin/env python3
import sys

def open_arg(path):
    return sys.stdin if path == '-' else open(path)

def strip_linebreak(s):
    return s[:-1] if s.endswith('\n') else s

with open_arg(sys.argv[1]) as pattern_file:
    patterns = set(map(strip_linebreak, pattern_file))

with open_arg(sys.argv[2]) as dataset_file:
    for l in map(strip_linebreak, dataset_file):
        if not patterns:



python3 contains-all.py file2 file1

The program exit status indicates whether all patterns of file 2 were matched:

  • 0 (success) means all patterns were matched.
  • 1 (failure) means some patterns weren't matched.

To query the exit status in a shell (script) you can either use the $? special variable or other expressions that evaluate command exit status, e. g. short circuit operators && and || and conditional expressions like if or while. Example:

if python3 compare-all.py file2 file1 && some-other --condition; then
    # do stuff

combine from moreutils will show you all the lines in file2 that aren't in file1 with:

combine file2 not file1

Then you can count the number of lines by piping it to wc -l, like:

if [ $(combine file2 not file1 | wc -l) != 0 ]; then
  echo "lines missing"
  echo "You're fine"

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .