How to get the difference between files

Question

I've found other links on the stackoverflow communities that were similar but they didn't answer my question exactly.

I have 2 files with a different number of lines BUT I have them both sorted. My original files are hundreds of lines long but for troubleshooting purposes, I made file1 have 12 lines and file2 have 5 lines. File2 is a subset of file1. What I want to do is run a command that outputs all the lines that are in file1 but are not in file2.

I tried using the Unix commands diff and comm but they both list the full contents of file1, which is not what I want.

A quick example of this would be:

File1      File2
A           B
B           E
C           I
E           N
G           O
I
L
M
N
O
X

So here, we can see everything that's in file2 is in file1. For some reason, diff and comm both showed the full contents of file1. I assume it's because it's doing a line by line comparison and not searching thru the whole file.

Is there another Unix command I can run that will output what I am expecting?

EDIT: The commands I used to attempt to get what I needed were:

a) diff file1 file2

This basically listed everything from file1 with a < in front of it showing the content was from file1, and everything from file2 with a > in front of it. Definitely not what I needed

b) comm -23 file1 file2

This showed the whole content of file1 again and not the diff like I was expecting. I also

c) comm -3 file1 file2

The help page for comm said this would print lines in file 1 but not in file 2 and vice versa but this also didn't show what I wanted b/c in my example, B appears in both files but on different lines. However, the output thinks it's in one but not the other and therefore prints it out. So the output looked like this:

A
B
    B
C
E
    E
etc.

And it wasn't what I was expecting. I was expecting

A
C
G
L
M
X

Answer 1

you can just reverse grep the content of file b in file A to do that

example :

francois@zaphod:/tmp$
 cat > A
az
ae
ar
at
ay
au
francois@zaphod:/tmp$
 cat > b
ba
by
ay
at
au
francois@zaphod:/tmp$
  grep -v "$(cat /tmp/b)" /tmp/A
az
ae
ar
francois@zaphod:/tmp$

here output is only line of file A not existing in file b

Answer 2

Diff output shows what has to be done that file X become file Y.

According to your comment, you might use the following oneliner:

cat x y | sort -u  | cat y - | sort | uniq -u

It will

1. Read both files
2. Resort them and print only unique lines (-u in sort)
3. Read the file (y) and combine it with piped output
4. Resort output
5. Use uniq so only lines that are presented once (-u in unique) are used.

This method is based on set theory. Firstly you add two sets, then subtract one of them.

Answer 3

If one of the files is a DOS formatted text file while the other file is a Unix formatted text file, then each line will be different from all lines in the other file, even if the letter on the lines are the same. This is due to the additional carriage-return character present at the end of each line in one file, which is not present in the other file.

I would expect the following output of comm -3 file1 file2 when file1 is a Unix text file while file2 is a DOS text file (this is what you have):

A
B
        B
C
E
        E
G
I
        I
L
M
N
        N
O
        O
X

I would expect the following output of comm -3 file1 file2 when file1 is a DOS text file while file2 is a Unix text file:

A
        B
B
C
        E
E
G
        I
I
L
M
        N
N
        O
O
X

If both files have the same line endings, i.e., they are both DOS text files or they are both Unix text files, I would expect the output of comm -3 file1 file2 to be

A
C
G
L
M
X

You may convert both files to Unix text format with the command

dos2unix file1 file2

... given that you have the dos2unix utility installed.