diff reports two files differ, although they are the same!

Question

I have two files which look identical to me (including trailing whitespaces and newlines) but diff still says they differ. Even when I do a (diff -y) side by side comparison the lines look exactly the same. The output from diff is the whole 2 files.

Any idea what's causing it?

Answer 1

Odd .. can you try cmp? You may want to use the '-b' option too.

cmp man page - Compare two files byte by byte.

This is one of the nice things about Unix/Linux .. so many tools :)

Answer 2

Might the differences be caused by DOS vs. UNIX line endings, or something similar?

What if you hexdump them? This might show differences more obviously, eg:

hexdump -C file1 > file1.hex
hexdump -C file2 > file2.hex
diff file1.hex file2.hex

Answer 3

My first guess, which turns out to be confirmed, is that the files use different line endings. It could be some other difference in whitespace, such as the presence of trailing whitespace (but you typically wouldn't get that on many lines) or different indentation (tabs vs spaces). Use a command that prints out whitespace and control characters in a visible form, such as

diff <(cat -A file1) <(cat -A file2)
diff <(sed -n l file1) <(sed -n l file2)

You can confirm that the differences only have to do with line endings by normalizing them first. You may have a dos2unix utility; if not, remove the extra CR (^M, \r, \015) character explicitly:

diff <(tr -d '\r' <file1) <(tr -d '\r' <file2)

or, if file1 is the one with DOS endings

 tr -d '\r' <file1 | diff - file2