Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

Is there any tool that can get lines which file A contains, but file B doesn't? I could make a little simple script with, e.g, perl, but if something like that already exists, I'll save my time from now on.

share|improve this question
    
add comment

6 Answers

up vote 33 down vote accepted

Yes. The standard grep tool for searching files for text strings can be used to subtract all the lines in one file from another.

grep -F -x -v -f fileB fileA

This works by using each line in fileB as a pattern (-f fileB) and treating it as a plain string to match (not a regular regex) (-F). You force the match to happen on the whole line (-x) and print out only the lines that don't match (-v). Therefore you are printing out the lines in fileA that don't contain the same data as any line in fileB.

The downside of this solution is that it doesn't take line order into account and if your input has duplicate lines in different places you might not get what you expect. The solution to that is to use a real comparison tool such as diff. You could do this by creating a diff file with the context value at 100% of the lines in the file, then parsing it for just the lines that would be removed if converting file A to file B. (Note this command also removes the diff formatting after it gets the right lines.)

diff -U $(wc -l < fileA) fileA fileB | grep '^-' | sed 's/^-//g' > fileC
share|improve this answer
    
@inderpreet99 The lower case -u argument does actually take a parameter of a number as long as it is not followed by a space. The advantage of the way I had it before is that it will work with or without a value, so you could use something in that sub command routine that returned not output. Upper case '-U' on the other hand requires an argument. –  Caleb Aug 27 '13 at 21:46
add comment

The grep and comm (with sort) methods take a long time on large files. SiegeX and ghostdog74 shared two great awk methods for extracting lines unique to one of two files over on Stack Overflow:

$ awk 'FNR==NR{a[$0]++}FNR!=NR && !a[$0]{print}' file1 file2

$ awk 'FNR==NR{a[$0]++;next}(!($0 in a))' file1 file2
share|improve this answer
add comment

The answer depends a great deal on the type and format of the files you are comparing.

If the files you are comparing are sorted text files, then the GNU tool written by Richard Stallman and Davide McKenzie called comm may perform the filtering you are after. It is part of the coreutils.

Example

Say you have the following 2 files:

$ cat a
1
2
3
4
5

$ cat b
1
2
3
4
5
6

Lines in file b that are not in file a:

$ comm <(sort a) <(sort b) -3
    6
share|improve this answer
    
+1 for mentioning comm; unfortunately, comm requires sorted files –  Arcege Feb 21 '12 at 4:04
4  
so sort them ? comm <(sort a) <(sort b) -1 -2 –  Sirex Feb 21 '12 at 8:17
add comment

from stackoverflow...

comm -23 file1 file2

-23 suppresses the lines that are in both files, or only in file 2. The files have to be sorted (they are in your example) but if not, pipe them through sort first...

See the man page here

share|improve this answer
add comment

If the files are big and you don't have a custom order to your entries, grep takes much too long. A quick alternative would be

sort file1 > 1 
sort file2 > 2 
diff 1 2 | grep "\>" | sed -e 's/> //'

[file2-file1 results to screen, pipe to file etc.]

Changing > to < would get the opposite subtraction. rm 1 2

share|improve this answer
add comment

You could also consider vimdiff, it highlights the differences between files in a vim editor

share|improve this answer
    
But is there an easy way to automatically do the subtraction in Vimdiff? –  Kazark Mar 14 '13 at 17:38
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.