112

Suppose there is a column of numeric values like following:

File1:

1 
2
3
3
3
4
4
4
5
6

I want the output:

3  
4

That is, only the repeated lines. Are there any command line tools to find this out in Linux? (NB: The values are numerically sorted).

2

4 Answers 4

174

You can use uniq(1) for this if the file is sorted:

uniq -d file.txt

If the file is not sorted, run it through sort(1) first:

sort file.txt | uniq -d

This will print out the duplicates only.

Technically the input does not need to be in sorted order, but the duplicates in the file need to be consecutive. The usual way to achieve that is to sort the file.

10
  • 1
    what if I want the triplicates only to be printed?
    – N. F.
    Oct 22, 2012 at 7:55
  • 11
    @MiNdFrEaK sort | uniq -c | grep '^\s*3\s' | sed 's/^\s*[0-9]*\s*//' for triplicates; replace "3" with any N for N-plicates Oct 22, 2012 at 8:10
  • 1
    @MiNdFrEaK sort | uniq -c | sed -n 's/^[[:blank:]]*3[[:blank:]]\{1,\}//p' for triplicates
    – user24222
    Oct 22, 2012 at 11:03
  • @camh can you do this on csv files as well? only values of a certain column? Jun 4, 2015 at 12:20
  • 2
    sort file.txt | uniq -d
    – ron
    Mar 30, 2017 at 8:31
7

uniq requires your list to be ordered, sort defaults to alphabetical

sort path/to/your/filename | uniq -d

or

cat fileName | sort | uniq -d

3

Execute this: perl -ne 'print if $a{$_}++' filename.txt

4
  • It gives 3\n3\n4\n\4n for the input File1 which is obviously wrong.
    – yaegashi
    Jul 10, 2015 at 0:03
  • the perl snip i find myself revisiting provides the number of incidences of each line so it can be piped, sorted, and filtered as needed: perl -ne '$a{$_}++; END { while(($k,$v)=each %a){printf "%d\t%s", $v,$k}}' filename Jun 2, 2016 at 22:00
  • Is there a way to do that on a specific column separated by a given field separator?
    – Geremia
    Sep 9, 2016 at 3:42
  • 1
    As indicated by yaegashi, a small fix is needed to fulfill the requirements: perl -ne 'print if 1==$a{$_}++' filename.txt Among all the answer, it is my favorite, because the other answers require to preprocess all the data with a full sort. This answer starts output results more quickly and efficiently.
    – BOC
    Jun 14, 2019 at 15:07
1

Using uniq and awk:

cat File1  | uniq -c | awk '$1 > 1 { print $2 }'
3
  • 7
    This work, but I don't see why you pipe the output of cat?
    – Bernhard
    Oct 22, 2012 at 17:52
  • 2
    not everyone knows you can do uniq -c File1 and similarly with many other tools. That is probably what is going on here.
    – Matthias
    Oct 24, 2016 at 17:15
  • 1
    Could still redirect uniq -c < File1. In particular, necessary with tr, as it does not process file args. Jul 6, 2020 at 23:02

Not the answer you're looking for? Browse other questions tagged or ask your own question.