40

Diff is a great tool to display the changes between two files. But how to display the similarities of two text files (while ignoring the differences)?

I.e. sample input:

a:
Foo Bar
X
Hello
World
42

b:
Foo Baz
Hello
World
23

Pseudo output (something like this):

@@ 2,3
=Hello World

Just sorting both files and using comm is not enough, because in that case the line information is lost.

2
  • You desire to throw away linewise information...by placing the output on one line? Assuming one-to-four words per line, that means you lose understand whether Hello World Mars Venus in your output originated on one line--or up to 4 different lines.
    – jubilatious1
    May 16 at 9:22
  • And what about duplicates? If Hello appears identically on four duplicated lines in each input file, do you want Hello to appear four times in your output?
    – jubilatious1
    May 16 at 9:25

5 Answers 5

Reset to default
37 
grep -Fxf file1 file2

-F means match plain strings (not regexps), -x means only whole-line matches, -f means take 'patterns' (i.e. lines) from the file named as its argument

3
  • 1
    This is not worked for me.
    – Chaminda Bandara
    Mar 8, 2019 at 11:45
  • @ChamindaBandara I’ve tried it with both the Mac’s built-in (POSIX/BSD) grep, where it fails, and with GNU grep, where it works as advertised. People tend to forget that the two are different.
    – Frungi
    Dec 30, 2020 at 0:53
  • @ajgringo and frungi standard grep accepts the given options, which means the suggested edit which proposed a "GNU grep" qualification would have made this answer worse rather than better.
    – Jeff Schaller ♦
    Dec 30, 2020 at 3:35
26

How about using diff, even though you don't want a diff? Try this:

diff --unchanged-group-format='@@ %dn,%df 
  %<' --old-group-format='' --new-group-format='' \
  --changed-group-format='' a.txt b.txt

Here is what I get with your sample data:

$ cat a.txt 
Foo Bar
X
Hello
World
42
$ cat b.txt 
Foo Baz
Hello
World
23
$ diff --unchanged-group-format='@@ %dn,%df
%<' --old-group-format='' --new-group-format='' \
  --changed-group-format='' a.txt b.txt
@@ 2,3
Hello
World
5
  • 3
    You can avoid embedding a literal newline like this: ...%df'$'\n''%<'...
    – Dennis Williamson
    Aug 24, 2010 at 2:54
  • 1
    You can also do it like this: ... --unchanged-group-format="@@ %dn,%df%c'\012'%<" ... (Note the double quotes.)
    – Dennis Williamson
    Aug 24, 2010 at 3:12
  • Great stuff! I did not know these options, because I just looked at the diff man page ...
    – maxschlepzig
    Aug 24, 2010 at 18:17
  • I'm using diff --version diff (GNU diffutils) 2.8.1 And I get the following error: diff: conflicting output style options diff: Try `diff --help' for more information.
    – Sujay
    Apr 21, 2014 at 12:09
  • 1
    I was getting "error: diff: conflicting output style options diff" because I had a diff alias defined. Use which diff to see if this is your problem.
    – justinjhendrick
    Feb 28, 2017 at 3:08
13

comm can be used. man comm for all the options but you'll want to use comm -12 ... to show only lines that exist in both inputs.

As people have pointed out, you need to pass your input through sort first.

4
  • 1
    Hm, that works only for common lines that are at the same line number in both files.
    – maxschlepzig
    Aug 23, 2010 at 15:02
  • 2
    comm seems to be for sorted files only and not to give that useful output for the usecase of the OP. His example: $ comm -12 a b Hello World comm: file 1 is not in sorted order comm: file 2 is not in sorted order
    – Marcel Stimberg
    Aug 23, 2010 at 15:04
  • @maxschlepzig: you should sort your files before passing them to comm.
    – Hemant
    Aug 23, 2010 at 15:05
  • 3
    By sorting you get rid of all information on the position of the common lines, though. You wouldn't sort files before comparing them with diff, either.
    – Marcel Stimberg
    Aug 23, 2010 at 15:17
8

I don't think there is a single command that does what you want it to do. You can try to combine the output of diff with grep, though. If your text files contain none of the characters |,<,>, the following gives you somewhat useful output:

$ diff --side-by-side a b | grep -n -v "[|<>]"
3:Hello                             Hello
4:World                             World
3
  • Try this: diff --width=155 --left-column --side-by-side a b | grep -n -v '|' | sed 's/ *($//'
    – Dennis Williamson
    Aug 24, 2010 at 2:48
  • that looks better -- but you have to include < and > in the grep to get also rid of added lines in either file.
    – Marcel Stimberg
    Aug 24, 2010 at 7:46
  • +1 For a simple, visual, and self-explanatory solution.
    – sema
    Apr 9, 2020 at 9:50
4

Dick Grune wrote a family of tools for this kind of thing:

http://dickgrune.com/Programs/similarity_tester/

There are versions that parse the syntax of various languages, so that things like renamed variables can be seen as unchanged.

It is packaged as similarity-tester in Debian and Ubuntu.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .