61

Suppose there is a column of numeric values like following:

File1:

1 
2
3
3
3
4
4
4
5
6

I want the output:

3  
4

That is, only the repeated lines. Are there any command line tools to find this out in Linux? (NB: The values are numerically sorted).

  • 3
    See man uniq. – jasonwryan Oct 22 '12 at 5:29
101

You can use uniq(1) for this:

uniq -d file.txt

This will print out the duplicates only. The input file needs to be sorted such that all duplicates are consecutive (which they appear to be), so run it through sort first if it is not.

  • 1
    what if I want the triplicates only to be printed? – MiNdFrEaK Oct 22 '12 at 7:55
  • 7
    @MiNdFrEaK sort | uniq -c | grep '^\s*3\s' | sed 's/^\s*[0-9]*\s*//' for triplicates; replace "3" with any N for N-plicates – full.stack.ex Oct 22 '12 at 8:10
  • @MiNdFrEaK sort | uniq -c | sed -n 's/^[[:blank:]]*3[[:blank:]]\{1,\}//p' for triplicates – user24222 Oct 22 '12 at 11:03
  • @camh can you do this on csv files as well? only values of a certain column? – NumenorForLife Jun 4 '15 at 12:20
  • 1
    sort file.txt | uniq -d – ron Mar 30 '17 at 8:31
1

Using uniq and awk:

cat File1  | uniq -c | awk '$1 > 1 { print $2 }'
  • 5
    This work, but I don't see why you pipe the output of cat? – Bernhard Oct 22 '12 at 17:52
  • not everyone knows you can do uniq -c File1 and similarly with many other tools. That is probably what is going on here. – Matthias Oct 24 '16 at 17:15
-1

Execute this: perl -ne 'print if $a{$_}++' filename.txt

  • It gives 3\n3\n4\n\4n for the input File1 which is obviously wrong. – yaegashi Jul 10 '15 at 0:03
  • the perl snip i find myself revisiting provides the number of incidences of each line so it can be piped, sorted, and filtered as needed: perl -ne '$a{$_}++; END { while(($k,$v)=each %a){printf "%d\t%s", $v,$k}}' filename – Theophrastus Jun 2 '16 at 22:00
  • Is there a way to do that on a specific column separated by a given field separator? – Geremia Sep 9 '16 at 3:42

protected by Stephen Kitt Jun 5 '16 at 12:13

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.