13

I got two files: file1 with about 10 000 lines and file2 with a few hundred lines. I want to check whether all lines of file2 occur in file1. That is: ∀ line ℓ ∈ file2 : ℓ ∈ file1

Should anyone not know what these symbols mean or what "check whether all lines of file2 occur in file1" means: Several equivalent lines in either files don't influence whether the check returns that the files meet the requirement or don't.

How do I do this?

  • 2
    May those files have duplicated lines? If file2 contains 2 lines A, do you need file1 to contain at least 2 lines A? – Stéphane Chazelas Oct 12 '17 at 19:57
  • 2
    @StéphaneChazelas All lines (in both files) are guaranteed to be unique. – UTF-8 Oct 12 '17 at 21:31
  • 1
    @UTF-8 That would be an important detail to edit into your question. – David Z Oct 13 '17 at 1:45
  • 2
    @DavidZ Not anymore as the existing answers don't rely on that guarantee. So by editing the question now, I'd reduce the apparent scope of the answers. – UTF-8 Oct 13 '17 at 10:08
  • @UTF-8 I suppose so, although the question is a little ambiguous without it, e.g. if a given line occurs 5 times in file2, does that line also have to occur 5 times in file1 (as opposed to only once)? If you had that requirement, it doesn't look like any of the existing answers would work, so I'd suggest at least editing in something that makes it clear that that's not what you mean. – David Z Oct 13 '17 at 18:06
16
comm -13 <(sort -u file_1) <(sort -u file_2)

This command will output lines unique to file_2. So, if output is empty, then all file_2 lines are contained in the file_1.

From comm's man:

   With  no  options,  produce  three-column  output.  Column one contains
   lines unique to FILE1, column two contains lines unique to  FILE2,  and
   column three contains lines common to both files.

   -1     suppress column 1 (lines unique to FILE1)

   -2     suppress column 2 (lines unique to FILE2)

   -3     suppress column 3 (lines that appear in both files)
  • @don_crissti True. Fixed: the -u option added to the sort command. Now, only unique lines are left in both sorted files. – MiniMax Oct 12 '17 at 19:58
  • Awesomely simply solution! Is this syntax applicable to any program which expects files? I always thought the < piped into stdin. Does the bracket term change this? – UTF-8 Oct 12 '17 at 21:43
  • 2
    @UTF-8 It is called Process substitution. You can read here about it. And yes, it behaves like a temporary file, so it can be used instead of real files in any programs, which expects files. – MiniMax Oct 12 '17 at 21:57
  • If this is something you do often, you might want to store file_1 in presorted form. Saves both typing and time. – Stig Hemmer Oct 13 '17 at 8:47
  • 7
    @minimax Good comment except for the "any". Process substitution, while wonderful, cannot be used in all cases, because the resulting "files" are streams and not real files. This means that they are not "seekable" like a normal file is, and can only be used when the program reads the file normally from the beginning, and not when the program uses some file-only functionality such as seeking to a specific point or rewinding to start over from the beginning. Happily, most programs simply read() their files, and so process substitution works with most programs, but not "any" programs. – Law29 Oct 13 '17 at 8:50
7
[ $(grep -cxFf file2 <(sort -u file1)) = $(sort -u file2 | wc -l) ] && 
  echo all there || 
  echo some missing

If the number of matches from file2 in (the unique lines of) file1 is the same as the number of unique lines in file2, then they're all there; otherwise, they aren't.

5

Using GNU awk where it does support specific length(array) feature (and some other awk implementation which may can support) and not required if files are sorted.

gawk 'FNR==NR{seen[$0];next} ($0 in seen){delete seen[$0]};
    END{print (!length(seen))?"Matched":"Not Matched"}' file2 file1

This is reading file2 into an array called seen with the key as entire line of file2.

Then read file1 and for each line if matched with lines in array seen then delete that key.

At the end if the array was empty means all lines in file2 exist in file1 and will print Matched, otherwise will display Not Matched.


For the compatibility in all awk implementations.

awk 'FNR==NR{seen[$0];next} ($0 in seen){delete seen[$0]};
    END{for(x in seen);print (!x)?"Matched":"Not Matched"}' file2 file1

To ignoring empty lines/or lines with whitespaces only if in file2, you would need to add NF to the condition in NR==FNR && NF {... to skip reading them into the array.

  • length(array) is gawk-only AFAIK; it definitely isn't POSIX. – dave_thompson_085 Oct 13 '17 at 4:15
  • @dave_thompson_085 Correct, I have updated my answer. thanks – αғsнιη Oct 13 '17 at 5:09
3

Using comm you can find lines that are common in both files.

comm -12 file1 file2

Have a look at man comm for more details

  • Correct it's returning common lines in both files, but this doesn't provide an answer to the OP's Q where if you had a line in file2 which doesn't exit in file1, so all lines of file2 doesn't exist in file1. – αғsнιη Oct 12 '17 at 18:12
  • 1
    files should be sorted. From man "comm - compare two sorted files line by line". – MiniMax Oct 12 '17 at 19:50
  • @MiniMax is right. This doesn't work. The other answer utilizing comm contains a solution which isn't obviously incorrect. When I run your command, I get warnings that the files aren't in sorted order and lots of lines which definitely are in both files. – UTF-8 Oct 12 '17 at 21:41
3
diff -q <(sort -u file2) <(grep -Fxf file2 file1 | sort -u)

will produce no output if file1 contains all the lines in file2 and exit with status 0, otherwise it will print something like

Files /proc/self/fd/11 and /proc/self/fd/12 differ

and exit with status 1

2

If I was only going to do it occasionally, I'd use a graphical diff viewer like meld. Open both files in it (meld file1 file2) and work your way down the shorter file, making sure all its lines exist in the other file. It would take a couple minutes but it's a nice visual way to convince yourself they're all there. It would work best if many or most of the lines will be grouped together in the larger file.

  • That's what I did when the smaller files only had 10 to 20 lines. But with hundreds, it's a lot of mundane work and this is Linux – there has to be a better solution. – UTF-8 Oct 12 '17 at 22:42
2

Use a Python program:

#!/usr/bin/env python3
import sys

def open_arg(path):
    return sys.stdin if path == '-' else open(path)

def strip_linebreak(s):
    return s[:-1] if s.endswith('\n') else s

with open_arg(sys.argv[1]) as pattern_file:
    patterns = set(map(strip_linebreak, pattern_file))

with open_arg(sys.argv[2]) as dataset_file:
    for l in map(strip_linebreak, dataset_file):
        patterns.remove(l)
        if not patterns:
            break

sys.exit(int(bool(patterns)))

Usage:

python3 contains-all.py file2 file1

The program exit status indicates whether all patterns of file 2 were matched:

  • 0 (success) means all patterns were matched.
  • 1 (failure) means some patterns weren't matched.

To query the exit status in a shell (script) you can either use the $? special variable or other expressions that evaluate command exit status, e. g. short circuit operators && and || and conditional expressions like if or while. Example:

if python3 compare-all.py file2 file1 && some-other --condition; then
    # do stuff
fi
1

combine from moreutils will show you all the lines in file2 that aren't in file1 with:

combine file2 not file1

Then you can count the number of lines by piping it to wc -l, like:

if [ $(combine file2 not file1 | wc -l) != 0 ]; then
  echo "lines missing"
else
  echo "You're fine"
fi

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.