110

Is there any tool that can get lines which file A contains, but file B doesn't? I could make a little simple script with, e.g, perl, but if something like that already exists, I'll save my time from now on.

159

Yes. The standard grep tool for searching files for text strings can be used to subtract all the lines in one file from another.

grep -F -x -v -f fileB fileA

This works by using each line in fileB as a pattern (-f fileB) and treating it as a plain string to match (not a regular regex) (-F). You force the match to happen on the whole line (-x) and print out only the lines that don't match (-v). Therefore you are printing out the lines in fileA that don't contain the same data as any line in fileB.

The downside of this solution is that it doesn't take line order into account and if your input has duplicate lines in different places you might not get what you expect. The solution to that is to use a real comparison tool such as diff. You could do this by creating a diff file with the context value at 100% of the lines in the file, then parsing it for just the lines that would be removed if converting file A to file B. (Note this command also removes the diff formatting after it gets the right lines.)

diff -U $(wc -l < fileA) fileA fileB | sed -n 's/^-//p' > fileC
  • @inderpreet99 The lower case -u argument does actually take a parameter of a number as long as it is not followed by a space. The advantage of the way I had it before is that it will work with or without a value, so you could use something in that sub command routine that returned not output. Upper case '-U' on the other hand requires an argument. – Caleb Aug 27 '13 at 21:46
  • be careful, grep -f is O(N^2) I believe: stackoverflow.com/questions/4780203/… – rogerdpack Oct 16 '15 at 17:33
  • 1
    the diff pipeline works a treat thanks. – Felipe Alvarez Nov 3 '16 at 23:01
  • To account for the sort problem, you could use process substitution in the command to process each file before the grep as needed. Example: grep -F -x -v -f <(sort fileB) <(sort fileA) – Tony Cesaro Oct 4 '17 at 15:22
  • @TonyCesaro That would work if your data set is not order specific and duplicates don't need to be taken into account. The advantage of using diff is that position in the file gets taken into account. – Caleb Oct 5 '17 at 6:06
57

The answer depends a great deal on the type and format of the files you are comparing.

If the files you are comparing are sorted text files, then the GNU tool written by Richard Stallman and Davide McKenzie called comm may perform the filtering you are after. It is part of the coreutils.

Example

Say you have the following 2 files:

$ cat a
1
2
3
4
5

$ cat b
1
2
3
4
5
6

Lines in file b that are not in file a:

$ comm <(sort a) <(sort b) -3
    6
  • 1
    +1 for mentioning comm; unfortunately, comm requires sorted files – Arcege Feb 21 '12 at 4:04
  • 11
    so sort them ? comm <(sort a) <(sort b) -1 -2 – Sirex Feb 21 '12 at 8:17
  • This is some weird syntax. <()? It works and I get it, but is there a name for this weirdness? – mlissner Apr 5 '17 at 22:47
  • 2
    @mlissner <() is also known as process substitution. – miku Apr 13 '17 at 15:51
  • 1
    comm was originally written circa 1973 by someone at Bell Labs, not rms. You're referring to the GNU implementation which came a lot later. There have been a lot of different implementations of the Unix utilities across the years. – Stéphane Chazelas Oct 30 '17 at 8:33
32

from stackoverflow...

comm -23 file1 file2

-23 suppresses the lines that are in both files, or only in file 2. The files have to be sorted (they are in your example) but if not, pipe them through sort first...

See the man page here

  • This doesn't work for me, for some reason ... – Jan Feb 7 '18 at 21:05
  • @Jan are your files sorted? How did you sort them? – JJS Feb 8 '18 at 19:03
8

The grep and comm (with sort) methods take a long time on large files. SiegeX and ghostdog74 shared two great awk methods for extracting lines unique to one of two files over on Stack Overflow:

$ awk 'FNR==NR{a[$0]++}FNR!=NR && !a[$0]{print}' file1 file2

$ awk 'FNR==NR{a[$0]++;next}(!($0 in a))' file1 file2
  • 2
    If you're doing this with huge files, then the memory constraints of loading a huge file into an associative array are going to be prohibitive. – Charles Duffy Jun 17 '16 at 14:01
4

If the files are big and you don't have a custom order to your entries, grep takes much too long. A quick alternative would be

sort file1 > 1 
sort file2 > 2 
diff 1 2 | grep "\>" | sed -e 's/> //'

[file2-file1 results to screen, pipe to file etc.]

Changing > to < would get the opposite subtraction. rm 1 2

2

You could also consider vimdiff, it highlights the differences between files in a vim editor

  • 1
    But is there an easy way to automatically do the subtraction in Vimdiff? – Kazark Mar 14 '13 at 17:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.