0

Previously, I asked the question: Identifying duplicate fields and print both with awk.

I have a file with multiple columns and wanted to identify lines where specific column values (cols 3-6) had been duplicated.

An answer to this was awk 'n=x[$3,$4,$5,$6]{print n"\n"$0;} {x[$3,$4,$5,$6]=$0;}' file

I now have the problem that I want to remove ALL lines identified using the above code from the datafile, leaving only lines that were never duplicated.

I have tried to use != instead of = but this gives the same result as = or returns 0 lines. I have also tried: awk '!seen[$3, $4, $5, $6]++' file but this also keeps the 1st instance of the duplicate that I want removed.

2

Although you are looking for a solution from awk, if your intended outcome is the elimination of your duplicates and not necessarily via awk alone, try:

  1. Firstly, ensure original input file is sorted, for example sort unsorted_file > file
  2. Run the awk command you previously found for identifying duplicates within columns 3-6, and saving output to a file, for example file_3-6_dupes, on your command prompt:
$ awk 'n=x[$3,$4,$5,$6]{print n"\n"$0;} {x[$3,$4,$5,$6]=$0;}' file > file_3-6_dupes
  1. Lastly, use comm to eliminate duplicates, saving the output to a file for example file_3-6_uniques:
$ comm -23 file file_3-6_dupes > file_3-6_uniques

How this works

  • a sorted input file is necessary because comm only works properly with sorted inputs
  • the awk command won't alter the order of appearance of the duplicates it discovers, it merely follows whatever order they were in the original file so really it is just the original file that needs to be sorted in the first place
  • by default comm outputs three columns: lines only in file 1, lines only in file 2, and lines in common
  • file 1: file
  • file 2: file_3-6_dupes
  • -number option specifies which of comm's output columns to suppress,
  • so -3 means, suppress comm's output column 3, what is common.
  • file_3-6_dupes which contains just duplicates, was derived from file, so these duplicates are the only things that are found in common to file and file_3-6_dupes
  • since we want the opposite of that, we just -3 to suppress what is common, which are the duplicates
  • on a minor note, we don't need an additional -2 to suppress things that are only in file 2, there are none, in our case

So by combining the use of awk, the original file, and comm, we can achieve your goal of eliminating lines with column 3-6 duplicates.

Tips

  • if the original file came from Windows, the non-Unix line endings may prevent comm from working properly with the awk-generated file_3-6_dupes, so you could run dos2unix on file if you find comm not working, and retry the steps, then it should work

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.