47

I have two big files (6GB each). They are unsorted, with linefeeds (\n) as separators. How can I diff them? It should take under 24h.

Improve this question

3 Answers 3

Reset to default
62

The most obvious answer is just to use the diff command and it is probably a good idea to add the --speed-large-files parameter to it.

diff --speed-large-files a.file b.file

You mention unsorted files so maybe you need to sort the files first

sort a.file > a.file.sorted
sort b.file > b.file.sorted
diff --speed-large-files a.file.sorted b.file.sorted

you could save creating an extra output file by piping the 2nd sort output direct into diff

sort a.file > a.file.sorted
sort b.file | diff --speed-large-files a.file.sorted -

Obviously these will run best on a system with plenty of available memory and you will likely need plenty of free disk space too.

It wasn't clear from your question whether you have tried these before. If so then it would be helpful to know what went wrong (took too long etc.). I have always found that the stock sort and diff commands tend to do at least as well as custom commands unless there are some very domain specific properties of the files that make it possible to do things differently.

Improve this answer
9
  • 2
    +1. You can omit all temporary files with named pipes. Use mkfifo to create [ab].file.sorted before using them as output for sort. Put both sorts with & in the background and use the both piped as filenames for diff.
    – krissi
    Sep 16, 2010 at 11:45
  • 15
    @krissi You can also accomplish the same effect using this syntax: diff <(command 1) <(command 2)
    – Michael Mrozek
    Sep 16, 2010 at 14:12
  • Thanks worked. I needed a couple of GB of memory thought but a 16GB Amazon EC2 instance fixed it :)
    – Jonas Lejon
    Sep 16, 2010 at 15:28
  • 7
    If someone like me wonders why <(cmd1) <(cmd2) syntax works (as it sounds like redirecting standard input twice!), try echo hello <(cmd1) <(cmd2). You'll see something like hello /dev/fd/63 /dev/fd/62 which suddenly makes it clear ;)
    – alex
    Sep 16, 2010 at 19:53
  • 4
    In my experience, the --speed-large-files option does not help if you do not have enough RAM. Also, pre-sorting is not helpful if you have a multi-line record structure you wish to preserve. The options referred to above (by @unhammer) are interesting, but the output from rdiff and bsdiff is rather binary. Installing bdiff from the Heirloom Toolbox looks like a dauning task (requires Heirloom devtools, extinct header files, …). Is it really worth the effort? Are there other alternatives?
    – Christian Pietsch
    Feb 2, 2015 at 17:33
8

Sorting the inputs and telling the diff program it's inputs are sorted would provide a massive speed up. I don't know of any diff with an option like that but comm assumes sorted input and will be much quicker if it does enough for your purposes.

Improve this answer
1
  • 3
    comm worked great for this, never heard of it before but apparently its in coreutils.
    – theferrit32
    Nov 16, 2019 at 6:14
0

The bdiff tool can work on unsorted files much larger than the computer's RAM.

Use these steps once, to download and compile bdiff before using it the first time:

wget https://github.com/Arkanosis/Arkonf/raw/master/tools-src/bdiff.c && \
  gcc -Wformat=0 -Wno-long-long bdiff.c -o bdiff && \
  rm bdiff.c

To run bdiff and compare 2 files:

./bdiff a.file b.file

You might find it helpful to redirect the bdiff output to a file. Thanks to @unhammer for the suggestion and the link to the Git repository.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.