0

I have two large files (2gig each) listed and sorted randomly, and I need to find difference between them and write to an output file:

file-1:

abc1234
def4321
ghi1234

file-2:

ghi1234
jk16754
abc1234
mnop3847

output:

def4321
jk16754
mnop3847
2
  • So you want to sort both files according to the same sorting order before diff?
    – njsg
    Aug 31, 2017 at 12:39
  • Do you want to select lines that are in file-1 and file-2 but lines that are in both files should be dropped?
    – mrc02_kr
    Aug 31, 2017 at 12:45

3 Answers 3

2
$ join -v 1 -v 2 <( sort file-1 ) <( sort file-2 )
def4321
jk16754
mnop3847

join performs a relational JOIN operation on its sorted input files (using the first tab-delimited column in each file, but this may be changed on the command line, see the manual). It outputs the matching data by default, but with -v 1 and -v 2 we may get the lines from each file that does not match in the other file.

If your shell does not support process substitution with <(...), then sort the files beforehand. You may want to do this anyway if you plan to run the command several times or do other work on the files.

$ sort -o file-1.sorted file-1
$ sort -o file-2.sorted file-2
$ join -v 1 -v 2 file-[12].sorted
0

A combination of comm and sed can do what you're looking for.

comm compares to files and outputs 3 columns. Column 1 is all the lines only in file1, column 2 is all the lines only in file2, and column 3 is common lines.

In your case, you want to suppress the printing of column 3, and then use sed to strip the whitespace from the entries in column 2.

bash-[533]$ comm -3 <(sort file-1) <(sort file-2) | sed -e 's/^\s*//'
def4321
jk16754
mnop3847

comm does require that the files are sorted, and the example above shows the files being sorted on the fly, with the output sent to a file descriptor, but you could pre-sort your files which may be easier on your system resources, and run the command as:

bash-[533]$ comm -3 file-1-sorted file-2-sorted | sed -e 's/^\s*//'
def4321
jk16754
mnop3847
1
  • The first two options worked for me: ` $ join -v 1 -v 2 <( sort file-1 ) <( sort file-2 )def4321 jk16754 mnop3847` AND diff option worked as well: ***diff --changed-group-format="%>%<" --unchanged-group-format="" <(sort file1) <(sort file2)
    – Riz
    Aug 31, 2017 at 17:42
0

With diff command:

diff --changed-group-format="%>%<" --unchanged-group-format="" <(sort file1) <(sort file2)

The output:

def4321
jk16754
mnop3847

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .