2

I have a file with multiple columns and want to identify those where specific column values (cols 3-6) have been duplicated.

The following code finds the duplicates but I want to display both instances, not just the second. The other column values (cols 1, 2 and 7+) can be different between the 2 lines hence the need for me to view both instances.

awk 'seen[$3, $4, $5, $6]++ == 1' filename
  • awk 'seen[$3, $4, $5, $6]++ >= 1' filename – cuonglm Aug 25 '15 at 17:06
  • 1
    Still only shows the second instance - not both – Bob Aug 25 '15 at 17:15
7

uniq is the correct tool for that:

uniq -D -f2 file

Where:

  • -D - prints all duplicates
  • -f2 - avoid comparing the first 2 fields

Edit: If the fields 7 and above are not to be compared, you need awk:

awk 'n=x[$3,$4,$5,$6]{print n"\n"$0;} {x[$3,$4,$5,$6]=$0;}' file
  • The array item x[] (columns 3-6) is checked. If it's already set run the part in {...} (in the same statement the n variable is set to the value of that array item)
  • In the brackets {...}: The n variable and the current line $0 are printed.
  • Then we set the x[] array item for the next iteration to the current line contents, for later comparsion.
  • Wouldn't -f2 still compare columns beyond col6? Also, what if I need to compare non-consecutive columns? – Bob Aug 25 '15 at 17:03
  • Doesn't it presuppose that the file is sorted on fields 3 to 6 also? – steeldriver Aug 25 '15 at 17:16
  • The edited version has done the job. Thanks @chaos! – Bob Aug 25 '15 at 17:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.