I have 2 genetic datasets. File1 is a txt file and has chromosome IDs which all also appear in file 2. File 2 is a csv file and is bigger than file 1 and has more unique chromosome IDs. How can I extract only the lines in file 2 which have unique chromosome IDs in comparison to file 1?

For example my data looks like this:

File 1 (chromosome position is actually my 125th column, implied by the ...):

Gene  pval    ... Chromosome position ID
ACE   0.002   ... 01:3290834_CT_C_1
NOS   0.01    ... 03:3304593_GA_G_1
BRCA  0.004 . ... 06:6265733_GA_G_1
CYP3  0.34    ... 09:9433933_GA_G_1

File 2 (chromosome position is my first column):

Chromosome position ID  Gene  pval
01:1243933_GA_G_1       ACE   0.002
03:3304593_GA_G_1       NOS   0.01
06:6265733_GA_G_1       BRCA  0.004
09:9433933_GA_G_1       CYP3  0.34

Based only on the chromosome position ID, the output unique line(s) which only appear in file 2 would be:

Chromosome position ID  Gene  pval
01:1243933_GA_G_1       ACE   0.002

I've tried to get this output with uniq and sort but currently I'm only finding examples online which use multiple text files but want the opposite where they take the matching ID lines not the unique ID lines. I have also tried grep but with these files being large the command is killed.

  • is this ask the same exercise you already asked in unix.stackexchange.com/questions/564514/…
    Commented Jan 31, 2020 at 14:38
  • Same data but instead of matching I am trying to find the unique values. Currently been trying to use that match finding code to apply to unique values only but I am struggling.
    – DN1
    Commented Jan 31, 2020 at 14:39

1 Answer 1


Adaption of the solution in your other thread (excluding the shell expansion file[12]):

awk ' FNR==NR {P[$125]; next} FNR==1 || !($1 in P)' file1 file2
Chromosome position ID  Gene  pval
01:1243933_GA_G_1       ACE   0.002
  • Thank you for this, at the moment it produces an empty file - one file is a csv and one is a txt file, would that cause this?
    – DN1
    Commented Jan 31, 2020 at 15:42
  • What is csv if not text? Don't your sample data represent the real data?
    – RudiC
    Commented Jan 31, 2020 at 16:14

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .