Efficient way to get the difference between two files in linux?

Question

I've been trying to get the difference between two huge .txt files (> 6Gb each) in linux. I have tried all the variants of diff, but the problem is that the process is soon aborted and ends with "memory exhausted" .

What I am looking to get is the exact difference between file1 and file 2 as:

file1.

aaa
bbb

file2.

bbb

difference (file1, file2):

 aaa

Any suggestion should take into account the huge size of the files to be compared (around 6Gb) and the limited memory available. Thank you very much

Answer 1

Since your question misses the important information:

• What do you understand by "all diff variants"?

• What file size do you have in mind and what number of lines do the files have?

the only way to answer your question is to give a lot of background information.

There are three basic diff implementations:

• The original UNIX diff based on the algorithm from Harold Stone. This usually comes in a herd together with diffh a simple diff for huge files and bdiff, a frontend to diff that calls diff with segmented parts of both files. This is the oldest diff implementation and a highly portable version of this herd of diff implementations is part of the schilytools.

• The programm udiff from the schilytools. This program is nearly 40 years old.

• GNU diff an implementation that is approx. 30 years old.

UNIX diff

is from the early 1970s and has been written for tiny machines with very limited memory. It still needs allocated RAM, typically 5*sizeof(int) * sum_of_lines.

The stone algorithm searches for the best resync match after an insertion or deletion and this consumes CPU time that is proportional to the square of the sum of the lines in the files. This limits the typical usability on recent hardware to a few megabytes per file until the diff operation takes too much time.

bdiff is a frontent for diff and automatically splits the files into chunks, runs diff with the chunks and finally combines the diff output. The resulting diff output is no longer optimal in case that a difference spans the position where files ar split. bdiff has been originally developped for SCCS in the late 1970s to speed up operation.

diffh is a separate program that uses CPU time that is approx. proportional to the file size but does an extremely halfhearted job for resyncing.

udiff

This program has been originally written in 1982. It produces an output that in most cases (if called as fdiff) is identical to the output from UNIX diff, but resyncing does not always works as nice as with the original diff. It consumes as much memory as the original UNIX diff but the consumed CPU time is only linear proportional to the size of the files.

GNU diff

Tries to give the same quality of resyncing as the original UNIX diff, but it consumes more RAM than the sum of the sizes of both files. This makes GNU diff a memory pig as it prevents it from being useful for huge files even though the CPU time consumption only increases linear with file size which makes GNU diff the best choice for mid size files of a few megabytes.

Recent versions of all of these programs have no line length limitation.

With typical file sizes, udiff (and it's sisters fdiff and fsdiff) are the fastest diff implementations. Since it does not try to slurp in both files (if they are larger than 64 MBytes) udiff is useful for huge files and it seems to be the best choice for your problem.

I recommend you to download a recent version of the schilytools from:

http://sourceforge.net/projects/schilytools/files/

compile, install and then run:

fdiff file1 file2