I am doing some sorting with uniq -d command. it's not getting me the desired results.

My file is like:

>TCONS_00000066 174     206  33
>TCONS_00000066 210     281  72
>TCONS_00000066 227     316  90
>TCONS_00000066 285     338  54
>TCONS_00000066 335     373  39
>TCONS_00000066 342     377  36
>TCONS_00006042 493     552  60
>TCONS_00006042 562     606  45
>TCONS_00006042 630     686  57
>TCONS_00006042 622     699  78
>TCONS_00006042 614     724  111
>TCONS_00006042 696     767  72

I want to remove duplicates in column 1



but I get a blank file everytime I do

uniq -d file > newfile

is there any other way?

  • 1
    You want to remove duplicates, or output duplicates? Your Question says remove but your example shows output. – roaima Mar 18 '15 at 9:40

You can use awk:

awk '!x[$1]++ { print $1 }' file

uniq -d lists only duplicated lines from its input; since your file contains no duplicate lines (considered in their entirety), it outputs nothing.

To get the result you're after, you need to extract the first column, and filter that:

cut -d\  -f1 file | uniq

assuming your file is sorted (uniq only filters duplicate adjacent lines); otherwise

cut -d\  -f1 file | sort -u

Or you can do it with awk as pointed out by taliezin, with a slight variant to only output the first column:

awk '!x[$1]++ { print $1 }' file

Uniq doesn't sort and it doesn't strip the rest of the line. uniq -d only shows duplicated lines, which is not the same as showing each line once (i.e. with -d it has to be in there twice, and consecutively to be shown)

First cut the non-interesting columns, then sort and then find the unique elements:

cut -d ' ' -f 1 your_file | sort | uniq

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.