3

I have a file A and B so I used to the following command...

(sort -n A B) | uniq -d

which should give me the numbers which occur in both files.

1
2
2
3
4
5
11
11
12
31

These are the numbers I get from sort -n A B but when I pipe it to uniq -d I only get 11 and not 2. What am I doing wrong?

  • Assuming the files are sorted, join A B is another option to acquire the intersection. – thrig May 24 '16 at 22:43
  • @don_crissti Thanks but I checked that already no trailing spaces. I even sent the output to a txt file still no trailing spaces.... – user2325601 May 24 '16 at 23:23
  • @don_crissti So what do I do to check or to avoid this? I manually checked the file there is nothing but you are right there is something. When I add a two consecutive 2's elsewhere it works. – user2325601 May 24 '16 at 23:38
  • Can either input file contain duplicates within itself? – glenn jackman May 25 '16 at 2:56
2

As the comments indicate, the problem seems likely to be blanks or carriage returns. Either of the following should do the trick:

$ (sort -n A B) | sed -E 's/[^[:alnum:]]+$//' | uniq -d
$ (sort -n A B) | tr -d '\r ' | uniq -d

Some flavors of GNU sed use -r instead to get Extended Regular Expressions. tr is certainly simpler but also more brutal in that it removes the characters whether or not they're trailing.

  • +1. With GNU sed, sed 's/\r$//' will strip trailing carriage-returns too. – cas May 25 '16 at 7:47
3

Because it's not used as much, I'll mention a -based solution:

comm -12 <(sort A) <(sort B)

This uses process substitution <( ... ) to sort files A and B and provide them as inputs to comm, which then uses -12 to:

  -1     suppress column 1 (lines unique to FILE1) 
  -2     suppress column 2 (lines unique to FILE2)

... leaving only lines common to both files.

2

In addition to what don_crissti mentions about trailing spaces you may want to check the filetype / newline style as well. The man page for uniq states that it can:

uniq - report or omit repeated lines

If you have say CRLF i.e. Windows style newline characters instead of the expected LF, you might get surprises.

You can quickly check the type with:

file <filename>

If you want to remove any CRLF endline sequences you can run the input files through dos2unix. The following will convert the end line characters.

dos2unix A
dos2unix B
  • It seems like you're on the right track. Filetype A and B are both CRLF and when I redirect them into a txt after running sort -n A B the filetype of that one is CRLF, LF. So what should I do in this case? – user2325601 May 24 '16 at 23:56
  • I would just run the A and B files through the dos2unix command and see if that helps. – C. Byrd May 24 '16 at 23:58
1

Depending on the file size you could just use grep:

grep -Fxf A B

-f specifies a file from which to get a list of patterns.

-x means to match the whole line only (disallow matching a part of a line).

-F means to treat the patterns as fixed strings rather than regular expressions.

If B is smaller than A you may get slightly faster results by naming B as the pattern file (grep -Fxf B A).

You can pipe the output to sort -u to get a sorted list of the distinct lines that occur in each file:

grep -Fxf A B | sort -u

Of course if your problem is carriage return line endings, you should use dos2unix first.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.