0

I have a tab delimited .txt file and want to delete rows that have duplications in the first column. If this can be done without sort, that would be amazing, but I've kind of come to terms with that I have to use sort. If I use sort, the header can not be sorted, it has to stay on top.

test dataset, (tab delimited):

Symbol  ATCACGACAGACAGGT.1  ATCACGACAGCCTATA.1  TTTGTCATCATGTCTT.1  
STPG1   0   1   3
NFYA    0   0   1
STPG1   1   3   1
ABD 0   0   0
ABC 0   0   0

My dream output:

Symbol  ATCACGACAGACAGGT.1  ATCACGACAGCCTATA.1  TTTGTCATCATGTCTT.1  
STPG1   0   1   3
NFYA    0   0   1
ABD 0   0   0
ABC 0   0   0

My next best output:

Symbol  ATCACGACAGACAGGT.1  ATCACGACAGCCTATA.1  TTTGTCATCATGTCTT.1  
ABC 0   0   0 
ABD 0   0   0   
NFYA    0   0   1
STPG1   0   1   3

I managed to use the following code on comma delimited test sample, testc.txt, except that the header is still in the sort. I can't seem to figure out a way to assign a tab delimiter that works? Also, having trouble with the header in this code as well:

sort -u -t, -k1,1 testc.txt

Maybe I should clarify that I do not know the number of rows, columns or what the row names are in my real data set.

2

You can use awk here:

$ awk -F'\t' 'NR==1 || !seen[$1]++' ip.txt
Symbol  ATCACGACAGACAGGT.1      ATCACGACAGCCTATA.1      TTTGTCATCATGTCTT.1
STPG1   0       1       3
NFYA    0       0       1
ABD     0       0       0
ABC     0       0       0
  • -F'\t' to specify tab as delimiter
  • NR==1 to preserve header (although for given input, this is not needed) - NR is a special variable that will have current line number
  • !seen[$1]++ here, the seen array will use first field as key and if that value is zero (first time seeing the key) then the condition will become true

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.