How can I print lines which have duplicate (x2) values ?


01 02 03
01 01 03
01 01 01 03

out of these three lines, only line two is correct.

so now let's say I want to look up for line where the value occurred x3.

in this case line 3 is correct.

  • When you’re looking for duplicates, does it matter which value is duplicated? For example, would 01 03 03 be valid? What about 01 01 03 03? May 12 '18 at 17:57
  • thank you for your response. No it does not matter which value is selected. And yes 01 03 03 is valid, and even 01 01 03 03.
    – Enrik S
    May 12 '18 at 18:01
  • @αғsнιη exactly with duplicate, it means x2. for 01 01 01 03, 01 occurs x3. The first answer is by way the easiest approach. And the 3rd answer is a great way to print more info about the results. Haven't tried the 2nd method yet. Thanks a lot for all the help guys.
    – Enrik S
    May 13 '18 at 11:58

With awk

awk -v nb=3 '{for(i=1;i<=NF;i++)if(++a[$i]>nb){print;next}}' infile

for(i=1;i<=NF;i++) on each field of the line
++a[$i] get each field in a associative array a and incremente it each time we saw a field with the same value.
if(++a[$i]>nb) if the value is more than nb
{print;next} print the line and jump to the next line

To show only the line with nb

awk -v nb=3 '
    max = 0
    delete a
    for ( i=1 ; i<=NF ; i++ )
    for( j in a )
        max = a[j]>max ? a[j] : max
    if ( max == nb )
}' infile
  • Been playing around with the coeff. Thank you it does work. Very clear explanation.
    – Enrik S
    May 12 '18 at 18:15
  • when i use nb=1, i get all the double even the triple ones, because of the (++a[$i]>nb), which implies that all doubles , triples or anythign higher is correct. But is there a way to limit it to only nb=1. Been trying to modify (++a[$i]>nb), but its not working.
    – Enrik S
    May 12 '18 at 18:30
  • Update the answer to limit
    – ctac_
    May 12 '18 at 20:04
  • 1
    See also split("", a) for a standard/portable equivalent of delete a. May 13 '18 at 10:48
  • 1
    @StéphaneChazelas thanks for tip, only man mawk give mawk supports an extension, delete array. Another way is to define a function getmax() with the array local.
    – ctac_
    May 13 '18 at 14:16

With AWK:

awk -v t=2 '{for (i=1; i<=NF; i++) c[$i]++; for (v in c) if (c[v] == t) {print; next}}'

This processes each line, and within each line, counts the occurrences of each value (each field) in the associative array c; then it goes over all the values v it has seen, and if one of the values was seen the required number of times (as specified by the target, t), it prints the line, and skips to the next line to avoid printing the line multiple times (e.g. for 01 01 03 03).

  • You'd need to empty the array (like with split("", c) or delete c with some implementations) between each record. May 13 '18 at 10:48

This will only print lines with duplicate space-delimited words:

while IFS='' read -r line ; do  if [[ "`echo $line | tr ' ' '\n'| sort | uniq -d`" != '' ]]; then echo "$line"; fi; done < YOURFILE

For your example the output will be:

01 01 03

01 01 01 03

Here, for the more than one occurrence of "01" in lines 2 and 3...

If you want to specify number of times a word should be checked to be repeated then :

NO=3; lnr=1 ; while IFS='' read -r line ; do echo "for line" $lnr ; echo $line | tr ' ' '\n' | uniq -c| grep -e "^\s*$NO" ; ((lnr++)); done < YOURFILE

For your example output will be:

for line 1

for line 2

for line 3

 3 01

The first number is the number of times you specified in variable NO for the number of occurrences to check.
The second number is the actual word that was found out to be repeating NO times.
Change YOURFILE to your file, of course.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.