79

Suppose there is a column of numeric values like following:

File1:

1 
2
3
3
3
4
4
4
5
6

I want the output:

3  
4

That is, only the repeated lines. Are there any command line tools to find this out in Linux? (NB: The values are numerically sorted).

131

You can use uniq(1) for this:

uniq -d file.txt

This will print out the duplicates only. The input file needs to be sorted such that all duplicates are consecutive (which they appear to be), so run it through sort first if it is not.

  • 1
    what if I want the triplicates only to be printed? – Nawshad Farruque Oct 22 '12 at 7:55
  • 10
    @MiNdFrEaK sort | uniq -c | grep '^\s*3\s' | sed 's/^\s*[0-9]*\s*//' for triplicates; replace "3" with any N for N-plicates – full.stack.ex Oct 22 '12 at 8:10
  • @MiNdFrEaK sort | uniq -c | sed -n 's/^[[:blank:]]*3[[:blank:]]\{1,\}//p' for triplicates – user24222 Oct 22 '12 at 11:03
  • @camh can you do this on csv files as well? only values of a certain column? – NumenorForLife Jun 4 '15 at 12:20
  • 1
    sort file.txt | uniq -d – ron Mar 30 '17 at 8:31
2

Execute this: perl -ne 'print if $a{$_}++' filename.txt

  • It gives 3\n3\n4\n\4n for the input File1 which is obviously wrong. – yaegashi Jul 10 '15 at 0:03
  • the perl snip i find myself revisiting provides the number of incidences of each line so it can be piped, sorted, and filtered as needed: perl -ne '$a{$_}++; END { while(($k,$v)=each %a){printf "%d\t%s", $v,$k}}' filename – Theophrastus Jun 2 '16 at 22:00
  • Is there a way to do that on a specific column separated by a given field separator? – Geremia Sep 9 '16 at 3:42
  • As indicated by yaegashi, a small fix is needed to fulfill the requirements: perl -ne 'print if 1==$a{$_}++' filename.txt Among all the answer, it is my favorite, because the other answers require to preprocess all the data with a full sort. This answer starts output results more quickly and efficiently. – BOC Jun 14 '19 at 15:07
2

uniq requires your list to be ordered, sort defaults to alphabetical

sort | uniq -d path/to/your/filename

or

cat fileName | sort | uniq -d path/to/your/filename

1

Using uniq and awk:

cat File1  | uniq -c | awk '$1 > 1 { print $2 }'
  • 7
    This work, but I don't see why you pipe the output of cat? – Bernhard Oct 22 '12 at 17:52
  • 2
    not everyone knows you can do uniq -c File1 and similarly with many other tools. That is probably what is going on here. – Matthias Oct 24 '16 at 17:15

Not the answer you're looking for? Browse other questions tagged or ask your own question.