Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I am using grep to filter out the contents on some patterns(genes in my case). For more info,here is the earlier link.

Find pattern from one file listed in another

My code(should work) but is not.

 grep -f file1 file2

Here is my subset of genes(file1):

C1QTNF3
C5orf22
C5orf28
C5orf34
C5orf38
C5orf42
C5orf49
C5orf51
C5orf64
C6
C7
C9
CAPSL
CARD6
CARTPT
CCDC125
CCDC152
CCL28
CCNB1
CCNO
CCT5
CD180
CDC20B
CDH10
CDH12
CDH18
CDH6
CDH9
CDK7
CENPH
CENPK
CKMT2
CLPTM1L
CMBL
CMYA5
COL4A3BP
CR749689
CRHBP
CRSP8P
CT49
CTNND2
CWC27
DAB2
DAP
DDX4
DEPDC1B
DHFR
DHX29
DIMT1
DMGDH

And below is my text file(file2) which is getting matched up,even though there is no gene UNC79 in file 1 as seen in SNPEFF_GENE_NAME=UNC79 show to be present in file2.

  AC=3;AF=0.016;AN=186;BaseQRankSum=0.075;DB;DP=292;Dels=0.00;FS=4.271;HaplotypeScore=0.0891;InbreedingCoeff=0.0225;MLEAC=2;MLEAF=0.011;MQ=59.18;MQ0=1;MQRankSum=0.969;QD=13.42;ReadPosRankSum=-0.373;SNPEFF_EFFECT=INTRON;SNPEFF_EXON_ID=23;SNPEFF_FUNCTIONAL_CLASS=NONE;SNPEFF_GENE_BIOTYPE=protein_coding;SNPEFF_GENE_NAME=UNC79;SNPEFF_IMPACT=MODIFIER;SNPEFF_TRANSCRIPT_ID=ENST00000256339;VQSLOD=9.31;culprit=DP

Hence,The output of grep is the whole text blob from file2.

Below is the complete row from the file,which gives issue.The second column is the gene name. I dont have this gene in my file1.And so I dont want the output of this particular row.I have 1000 such rows of different genes,which need to be filtered out only for the genes that are in file1.

    intronic    UNC79   14  94062922    94062922    A   G   het 80.54   3   14  94062922    rs183710732 A   G   80.54   PASS    AC=3;AF=0.016;AN=186;BaseQRankSum=0.075;DB;DP=292;Dels=0.00;FS=4.271;HaplotypeScore=0.0891;InbreedingCoeff=0.0225;MLEAC=2;MLEAF=0.011;MQ=59.18;MQ0=1;MQRankSum=0.969;QD=13.42;ReadPosRankSum=-0.373;SNPEFF_EFFECT=INTRON;SNPEFF_EXON_ID=23;SNPEFF_FUNCTIONAL_CLASS=NONE;SNPEFF_GENE_BIOTYPE=protein_coding;SNPEFF_GENE_NAME=UNC79;SNPEFF_IMPACT=MODIFIER;SNPEFF_TRANSCRIPT_ID=ENST00000256339;VQSLOD=9.31;culprit=DP    GT:AD:DP:GQ:PL  0/1:1,2:3:33:39,0,33
share|improve this question

3 Answers 3

up vote 5 down vote accepted

Since your gene names are always in the 2nd column of the file, you can use awk for this:

awk '
    {   ## while reading the first file, save name in the array a
        if(NR==FNR){a[$1]++;} 

        ## If this is the 2nd file
        else{
            ## print if the value of the second column is defined in the array 
            if($2 in a){print}
        }
    }' file1 file2

The same, condensed:

awk '{if(NR==FNR){a[$1]++;}else{if($2 in a){print}}}' file1 file2 

more condensed:

awk '(NR==FNR){a[$1]++}($2 in a){print}' file1 file2 

and truly minimalist (in answer to @Awk):

awk 'NR==FNR{a[$1]}$2 in a' file1 file2 
share|improve this answer
    
+1 you can make even simpler using next and a[$1], is just enough. a[$1]++ holds count of occurrence of index that is $1 –  Akshay Hegde Feb 20 at 17:07
    
@Awk the next is irrelevant, I added a truly minimalist version for your pleasure :). –  terdon Feb 20 at 17:13
    
Isn't grep able to restrict it to only a particular column,instead of searching through whole file?I was just curious on this.I tried to find something but wasn't. –  Ron Feb 21 at 5:09
1  
@Ron no, grep does not know about columns, it just sees a stream of bytes. Whenever you want to deal with columns, think awk. –  terdon Feb 21 at 5:16

Your file2 includes a substring that grep is matching with one of the lines from file1.

Example

                      ss #1

You can see it in the above screenshot, the text is highlighted in red by grep. Incidentally you might want to use the color highlighting feature too:

$ grep --color=auto -f file1 file2

But I want to match only whole words??

If you want grep to only return matches that are "whole words" you can include the -w switch. This will only return matches are whole word matches using the "words" from file1.

Here I've created another file (file1a) that does include the gene UNC79.

$ grep C7 file1 file1a
file1:C7
file1a:C7
file1a:UNC79

Here when I run the grep -wf ... command with the 2 index files (file1 and file1a) you can see that we get no match for file1 and a match with file1a.

           ss #2

excerpt from grep man page

   -w, --word-regexp
          Select only those lines containing matches that form whole words.
          The test is that the matching substring must  either  be  at  the
          beginning  of  the  line,  or  preceded by a non-word constituent
          character.  Similarly, it must be either at the end of the line or
          followed by a non-word constituent character.  Word-constituent
          characters are letters, digits, and the underscore.

This trick works in @Ron's situation because his gene names are bordered by non-word characters (=) and they're terminated with (;). Otherwise this trick would not likely work.

share|improve this answer
    
I just want only the content if it matches exactly.e.g. if I had this gene name UNC79 in file1 ,then i want the whole content from file 2. –  Ron Feb 19 at 23:05
    
I tried " grep -F -f file1 file2" ,as -F is used for fixed strings to be matched. –  Ron Feb 19 at 23:07
    
@Ron - I think you're thinking about the grep feature in the wrong way. If you told grep to find strings that were present in file1 how do you expect it to anchor the matches that it finds? Why would it not return C7 as a match but would return UNC79? What is achoring UNC79 that would keep it from not matching XXUNC79XX too? –  slm Feb 19 at 23:15
    
@Ron - thinking of it another way, when grep encounters C7 in your current file, what is special about it that would make grep not return it as a match? –  slm Feb 19 at 23:17
1  
@Ron - here's a way to do what you want. Use grep -wf file1 file2. –  slm Feb 19 at 23:33

@ terdon You can make more simple

awk 'NR==FNR{a[$1];next}($2 in a)' fil1 file2

--edit---

@terdon

+1 if always it's in 2nd column then fine, but still problem is after storing index in array that is a[$1] while reading file1, it simply checks $2 in a of file1, this just kills some time also, here is example of getting unexpected result without next, please don't mind here I am not telling your answer is wrong, I am just trying to show you that it simply reads file even after its work already over that is storing index in array a.

$ cat f1
1 0
2 0
3 0
4 1
1 2
2 3
3 4

$ cat f2
NOTHING

# Bad result without next, still awk is reading f1, this kills time right ?
$ awk 'NR==FNR{a[$1]}$2 in a' f1 f2
4 1
1 2
2 3
3 4


# It gives nothing but it's right 
$ awk 'NR==FNR{a[$1];next}$2 in a' f1 f2
share|improve this answer
1  
Nice, +1 but as I said above, the next is not needed. Nor the parentheses if you want to get golfish :) –  terdon Feb 20 at 17:14
    
You're quite right, using next is safer, even though it's not needed in the specific case. Good catch. –  terdon Feb 21 at 5:18

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.