Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

Is there a unix command that can check if any two lines in a file are the same?

For e.g. Consider a file sentences.txt

This is sentence X
This is sentence Y
This is sentence Z
This is sentence X
This is sentence A
This is sentence B

We see that the sentence

This is sentence X

is repeated.

Is there any command that can quickly detect this, so that I can perhaps execute it like this -

$ cat sentences.txt | thecommand
Line 1:This is sentence X
Line 4:This is sentence X
share|improve this question
add comment

3 Answers

up vote 35 down vote accepted

Here is one way to get the exact output you're looking for:

$ grep -nFx "$(sort sentences.txt | uniq -d)" sentences.txt 
1:This is sentence X
4:This is sentence X

Explanation:

The inner $(sort sentences.txt | uniq -d) lists each line that occurs more than once. The outer grep -nFx looks again in sentences.txt for exact -x matches to any of these lines -F and prepends their line number -n

share|improve this answer
    
Your edit just barely beat me from posting the exact same answer. +1 –  casey Feb 5 at 18:40
    
+1 for giving the exact output in a really nice way! –  aularon Feb 5 at 18:42
    
Thanks! This is a really informative answer. –  CodeBlue Feb 5 at 19:26
1  
@CodeBlue - yes. It's called Command Substitution –  grebneke Feb 5 at 19:29
5  
sort sentences.txt | uniq -d | grep -nFxf - sentences.txt would be a little more efficient and would avoid potential arg list too long problems. –  Stephane Chazelas Feb 6 at 9:34
show 2 more comments

Not exactly what you want, but you can try combining sort and uniq -c -d:

aularon@aularon-laptop:~$ cat input
This is sentence X
This is sentence Y
This is sentence Z
This is sentence X
This is sentence A
This is sentence B

aularon@aularon-laptop:~$ sort input | uniq -cd
      2 This is sentence X
aularon@aularon-laptop:~$ 

2 here is the number of duplications found for the line, from man uniq:

   -c, --count
          prefix lines by the number of occurrences

   -d, --repeated
          only print duplicate lines
share|improve this answer
add comment

IF the file contents fit in memory awk is good for this. The standard one-liner in comp.lang.awk (I can't search an instance from this machine but there's several every month) to just detect there is duplication is awk 'n[$0]++' which counts the occurrences of each line value and prints any occurrence(s) other than the first, because the default action is print $0.

To show all occurrences including the first, in your format, but possibly in mixed order when more than one value is duplicated, gets a little more finicky:

awk <sentences.txt ' !($0 in n) {n[$0]=NR;next} \
    n[$0] {n[$0]=0; print "Line "n[$0]":"$0} \
    {print "Line "NR":"$0} '

Shown in multiple lines for clarity, you usually run together in real use. If you do this often you can put the awk script in a file with awk -f, or of course the whole thing in a shell script. Like most simple awk this can be done very similarly with perl -n[a].

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.