My file is semicolon separated file with multiple columns. I want to remove duplicate records from first column only. Also, i want a count of these duplicate entries from column first only. any suggestion please?

  • 3
    Could you edit your question to show some sample input and output, at least a mock-up of the general structure? – ilkkachu Apr 19 '17 at 10:50

Something like this should work:

awk -F";" '!seen[$1]++{print}END {for (i in seen) if (seen[i]>1) print i,"found :" seen[i] "times"}' file

The first part before END will not print duplicates emulating uniq. That means tha if an entry is found two or more times only the first entry will be printed.

The code in END will print all entries found more than once.

If you need to print pure unique entries emulating uniq -u meaning tha if an entry is found two or more times will not be printed at all then you may use:

awk -F";" '{seen[$1]++}END {for (k=1;k<=2;k++) for (i in seen) if (seen[i]==1) {print i;delete seen[i];continue} else {print i,"found :" seen[i] "times"}}' file

This will iterate the array two times. The first time true uniq entries will be printed and the second time duplicate entries will be printed.


awk -F";" '{seen[$1]++}END {for (i in seen) print "found :" seen[i] "times:",i}' file |sort

This will print each entry along with the counting (unique entries will be found once, non unique entries will be found more than once)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.