Is there a tool to subtract one file from another?

Question

Is there any tool that can get lines which file A contains, but file B doesn't? I could make a little simple script with, e.g, perl, but if something like that already exists, I'll save my time from now on.

Answer 1

Yes. The standard grep tool for searching files for text strings can be used to subtract all the lines in one file from another.

grep -F -x -v -f fileB fileA

This works by using each line in fileB as a pattern (-f fileB) and treating it as a plain string to match (not a regular regex) (-F). You force the match to happen on the whole line (-x) and print out only the lines that don't match (-v). Therefore you are printing out the lines in fileA that don't contain the same data as any line in fileB.

The downside of this solution is that it doesn't take line order into account and if your input has duplicate lines in different places you might not get what you expect. The solution to that is to use a real comparison tool such as diff. You could do this by creating a diff file with the context value at 100% of the lines in the file, then parsing it for just the lines that would be removed if converting file A to file B. _{(Note this command also removes the diff formatting after it gets the right lines.)}

diff -u$(wc -l < fileA) fileA fileB | grep '^-' | sed 's/^-//g'

Answer 2

If the files are big and you don't have a custom order to your entries, grep takes much too long. A quick alternative would be

sort file1 > 1 
sort file2 > 2 
diff 1 2 | grep "\>" | sed -e 's/> //'

[file2-file1 results to screen, pipe to file etc.]

Changing > to < would get the opposite subtraction. rm 1 2

Answer 3

You could also consider vimdiff, it highlights the differences between files in a vim editor

Answer 4

The answer depends a great deal on the type and format of the files you are comparing.

If the files you are comparing are sorted text files, then the GNU tool written by Richard Stallman and Davide McKenzie called comm may perform the filtering you are after. It is part of the coreutils.