1

I've found other links on the stackoverflow communities that were similar but they didn't answer my question exactly.

I have 2 files with a different number of lines BUT I have them both sorted. My original files are hundreds of lines long but for troubleshooting purposes, I made file1 have 12 lines and file2 have 5 lines. File2 is a subset of file1. What I want to do is run a command that outputs all the lines that are in file1 but are not in file2.

I tried using the Unix commands diff and comm but they both list the full contents of file1, which is not what I want.

A quick example of this would be:

File1      File2
A           B
B           E
C           I
E           N
G           O
I
L
M
N
O
X

So here, we can see everything that's in file2 is in file1. For some reason, diff and comm both showed the full contents of file1. I assume it's because it's doing a line by line comparison and not searching thru the whole file.

Is there another Unix command I can run that will output what I am expecting?

EDIT: The commands I used to attempt to get what I needed were:

a) diff file1 file2

This basically listed everything from file1 with a < in front of it showing the content was from file1, and everything from file2 with a > in front of it. Definitely not what I needed

b) comm -23 file1 file2

This showed the whole content of file1 again and not the diff like I was expecting. I also

c) comm -3 file1 file2

The help page for comm said this would print lines in file 1 but not in file 2 and vice versa but this also didn't show what I wanted b/c in my example, B appears in both files but on different lines. However, the output thinks it's in one but not the other and therefore prints it out. So the output looked like this:

A
B
    B
C
E
    E
etc.

And it wasn't what I was expecting. I was expecting

A
C
G
L
M
X
6
  • Diff output shows what have to be done that file X become file Y. If I get you correctly you want to know all lines that are in FILE1 but are not in FILE2? Nov 2, 2021 at 22:45
  • 2
    It may be of help if you could add the exact commands you tried to your question. For instance, given your sample data, comm -2 -3 File1 File2 should output A C G L M X. If it does not, there may be something unexpected going on with the data, e.g. CRLF (Windows) line terminators in one of the two files.
    – fra-san
    Nov 2, 2021 at 22:51
  • 1
    @fra-san thanks, THAT was the problem. There was a space that was there that messed me up. :(
    – Classified
    Nov 2, 2021 at 23:23
  • 1
    Read man diff. Everything you've asked is covered there.
    – Seamus
    Nov 3, 2021 at 5:33
  • 2
    Thanks, @Classified. I am voting to close this question because it originated from a peculiarity in the data that is not shown here (and hence the issue can not be reproduced by other users).
    – fra-san
    Nov 3, 2021 at 9:30

3 Answers 3

0

you can just reverse grep the content of file b in file A to do that

example :

francois@zaphod:/tmp$
 cat > A
az
ae
ar
at
ay
au
francois@zaphod:/tmp$
 cat > b
ba
by
ay
at
au
francois@zaphod:/tmp$
  grep -v "$(cat /tmp/b)" /tmp/A
az
ae
ar
francois@zaphod:/tmp$

here output is only line of file A not existing in file b

1
  • 1
    The way you use grep here will treat each line of b as a regular expression. It would be better to use grep with both its -x and -F options.
    – Kusalananda
    Apr 18 at 7:30
0

Diff output shows what has to be done that file X become file Y.

According to your comment, you might use the following oneliner:

cat x y | sort -u  | cat y - | sort | uniq -u

It will

  1. Read both files
  2. Resort them and print only unique lines (-u in sort)
  3. Read the file (y) and combine it with piped output
  4. Resort output
  5. Use uniq so only lines that are presented once (-u in unique) are used.

This method is based on set theory. Firstly you add two sets, then subtract one of them.

0

If one of the files is a DOS formatted text file while the other file is a Unix formatted text file, then each line will be different from all lines in the other file, even if the letter on the lines are the same. This is due to the additional carriage-return character present at the end of each line in one file, which is not present in the other file.

I would expect the following output of comm -3 file1 file2 when file1 is a Unix text file while file2 is a DOS text file (this is what you have):

A
B
        B
C
E
        E
G
I
        I
L
M
N
        N
O
        O
X

I would expect the following output of comm -3 file1 file2 when file1 is a DOS text file while file2 is a Unix text file:

A
        B
B
C
        E
E
G
        I
I
L
M
        N
N
        O
O
X

If both files have the same line endings, i.e., they are both DOS text files or they are both Unix text files, I would expect the output of comm -3 file1 file2 to be

A
C
G
L
M
X

You may convert both files to Unix text format with the command

dos2unix file1 file2

... given that you have the dos2unix utility installed.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.