Find text not in both files using grep

Question

I want to search two text files and print to a file the lines of each file that are only in one of the file.

For example, LIST-1.txt contains:

apples
dogs
paintings
mom
dad
don

LIST-2.txt contains

apples
don
dad
mom
cats

I want the output to be (in any order):

dogs
paintings
cats

I have tried this:

cat list-1.txt | while read line || [[ -n $line ]];
do
   grep -v $line list-2.txt
done

Any suggestions?

Answer 1

The issue with your loop is that in each iteration, you will get all lines in the second file that is not the same as the current line in the first file.

A variation of your loop is the concatenation of the output of the following two commands:

grep -v -xF -f LIST-1.txt LIST-2.txt
grep -v -xF -f LIST-2.txt LIST-1.txt

The first grep will get all lines from LIST-2.txt that does not correspond exactly to any line in LIST-1.txt, while the second grep does the same thing with the two files reversed. This would actually give you the result that you mention in the question. (I see that Jeff already mentioned this, so if you like this approach, go upvote his answer, not mine.)

It does require reading in one of the files into memory (as query strings), and would possible be considered slightly inelegant. I also haven't really thought about under what circumstances it may fail to provide correct result.

Personally, I would go with

$ join -v 1 -v 2 <( sort LIST-1.txt ) <( sort LIST-2.txt )
cats
dogs
paintings

This performs a relational JOIN operation between the files. Normally, this would return the entries that exists in both files (an inner join), but here we ask with -v 1 -v 2 to see all entries that don't match up in either file.

The join utility requires sorted input (to be able to only hold one line from each file in memory at a time), which is why we sort both files and provide them to join via individual process substitutions.

In shells that do not have process substitutions, you may want to create sorted copies of the files before calling join:

sort -o LIST-1.txt.sorted LIST-1.txt &&
sort -o LIST-2.txt.sorted LIST-2.txt &&
join -v 1 -v 2 LIST-[12].txt.sorted
rm -f LIST-[12].txt.sorted

Answer 2

Since it was just mentioned, but never explained explicitly: a GNU comm solution

comm --output-delimiter '' -3 <(sort file1) <(sort file2)

-3 suppresses lines that appear in both files, and the delimiter specification will just left-align the result. However the files need to be sorted for comm to be working.

Answer 3

If you don't care about the results being re-sorted, one approach could be to:

1. sort the two files together
2. ask uniq to count the number of adjacent duplicates
3. ask awk to print only the lines that are not duplicated
4. ask cut to remove uniq's counts

sort list-1.txt list-2.txt | uniq -c | awk '$1 == 1 { print}' | cut -c9-

To force grep to do it, you could run a whole-line, fixed-text, file-based exclusion in both directions:

{ grep -vxF -f list-1.txt list-2.txt; grep -vxF -f list-2.txt list-1.txt; }

This asks grep for the lines in the 2nd file that do not exist in the 1st file, where we reverse the filenames for the 2nd grep.

Answer 4

if your input files have no duplicated entries individually, you can use:

sort list[12] |uniq -u

or use awk as an alternative:

awk '{ seen[$0]++ } END{ for (x in seen) if (seen[x]==1) print x }' list[12]