Identify strings between patterns and print entire region between pattern if string is found. Perferably using awk

Question

There are similar questions but none of them quite address the issue I have.

Simply put, I need to print every block that contains any string I'm looking for. Each block start line contains: <entry version=

For more detail see below:

I want to search a massive file (hundreds of thousands of lines) printing each entire region (block) between patterns if a certain string is identified within the pattern region (block).

I get that I can print entire regions between patterns, where start and end identifiers for these blocks are "/<entry version=" using: awk '/<entry version=/{flag=1} flag; /<entry version=/{flag=0}'

but how do I get it to only print the entire block if certain strings are found between these patterns?

Shortest part of the real data looks like this for a block region (though in reality each block is thousands of lines long), I want to thank Terdon for sorting a better example for me to use:

<entry version="21.1" url="http://v21.proteinatlas.org/ENSG00000000003">   
        <name>TSPAN6</name>                                                                                                                             
        <synonym>T245</synonym>
        <synonym>TM4SF6</synonym>
        <synonym>TSPAN-6</synonym>
        <identifier id="ENSG00000000003" db="Ensembl" version="103.38" assembly="GRCh38.p13" gencodeVersion="37">
                <xref id="O43657" db="Uniprot/SWISSPROT"/> 
                <xref id="7105" db="NCBI GeneID"/>
        </identifier>  
        <proteinClasses>   
                <proteinClass source="MDM" id="Ma" parent_id="" name="Predicted membrane proteins"/>

In the above real format, I would be checking the name and synonyms for specific strings, so if I was looking for "TSPAN6" then this block would be printed. The blocks are thousand of lines each so below is just a made up mini version to explain how I want to print blocks based on string matches within the block.

Here is an example, if my strings were "MEMSAT" and "TNMD"
Example input:

 <entry version="21.1" url="http://v21.proteinatlas.org/ENSG00000000003">
        <name>TSPAN6</name>
        <synonym>T245</synonym>
        <synonym>TM4SF6</synonym>
        <synonym>TSPAN-6</synonym>
        <identifier id="ENSG00000000003" db="Ensembl" version="103.38" assembly="GRCh38.p13" gencodeVersion="37">
            <xref id="O43657" db="Uniprot/SWISSPROT"/>
            <xref id="7105" db="NCBI GeneID"/>
        </identifier>
        <proteinClasses>
            <proteinClass source="MDM" id="Ma" parent_id="" name="Predicted membrane proteins"/>
            <proteinClass source="Ezkurdia et al 2014" id="Eb" parent_id="" name="Protein evidence (Ezkurdia et al 2014)"/>
        </proteinClasses>
        <proteinEvidence evidence="Evidence at protein level">
            <evidence source="HPA" evidence="Evidence at transcript level"/>
            <evidence source="MS" evidence="Not available"/>
            <evidence source="UniProt" evidence="Evidence at protein level"/>
        </proteinEvidence>
  </entry>
    <entry version="21.1" url="http://v21.proteinatlas.org/ENSG00000000005">
        <name>TNMD</name>
        <synonym>BRICD4</synonym>
        <synonym>ChM1L</synonym>
        <synonym>myodulin</synonym>
        <synonym>TEM</synonym>
        <synonym>tendin</synonym>
        <identifier id="ENSG00000000005" db="Ensembl" version="103.38" assembly="GRCh38.p13" gencodeVersion="37">
            <xref id="Q9H2S6" db="Uniprot/SWISSPROT"/>
            <xref id="64102" db="NCBI GeneID"/>
        </identifier>
        <proteinClasses>
            <proteinClass source="MDM" id="Ma" parent_id="" name="Predicted membrane proteins"/>
            <proteinClass source="MDM" id="Md" parent_id="" name="Membrane proteins predicted by MDM"/>
            <proteinClass source="MEMSAT3" id="Me" parent_id="" name="MEMSAT3 predicted membrane proteins"/>
        </proteinClasses>
        <proteinEvidence evidence="Evidence at protein level">
            <evidence source="HPA" evidence="Evidence at transcript level"/>
            <evidence source="MS" evidence="Not available"/>
            <evidence source="UniProt" evidence="Evidence at protein level"/>
        </proteinEvidence>
  </entry>

Example output:

    <entry version="21.1" url="http://v21.proteinatlas.org/ENSG00000000005">
        <name>TNMD</name>
        <synonym>BRICD4</synonym>
        <synonym>ChM1L</synonym>
        <synonym>myodulin</synonym>
        <synonym>TEM</synonym>
        <synonym>tendin</synonym>
        <identifier id="ENSG00000000005" db="Ensembl" version="103.38" assembly="GRCh38.p13" gencodeVersion="37">
            <xref id="Q9H2S6" db="Uniprot/SWISSPROT"/>
            <xref id="64102" db="NCBI GeneID"/>
        </identifier>
        <proteinClasses>
            <proteinClass source="MDM" id="Ma" parent_id="" name="Predicted membrane proteins"/>
            <proteinClass source="MDM" id="Md" parent_id="" name="Membrane proteins predicted by MDM"/>
            <proteinClass source="MEMSAT3" id="Me" parent_id="" name="MEMSAT3 predicted membrane proteins"/>
        </proteinClasses>
        <proteinEvidence evidence="Evidence at protein level">
            <evidence source="HPA" evidence="Evidence at transcript level"/>
            <evidence source="MS" evidence="Not available"/>
            <evidence source="UniProt" evidence="Evidence at protein level"/>
        </proteinEvidence>
  </entry>

Can we assume you're browsing/downloading data from one of the links here? proteinatlas.org/about/download — jubilatious1, Commented Oct 20, 2022 at 15:33
yes, specifically proteinatlas.org/download/proteinatlas.xml.gz but its huge so I tried to simplify and present something small — Giles, Commented Oct 20, 2022 at 15:34
I made a small toy example by just taking the first two entries of the protein atlas data, removing most of their lines, and adding the necessary closing tags. You can see it here: pastebin.com/iFQhQzhK. Could you please take that example and then provide a string to look for and the output you would expect from that example? Just edit your question to use that instead of the original. That way, we can give you proper solutions. — terdon, Commented Oct 20, 2022 at 16:06
Comments are not for extended discussion; this conversation has been moved to chat. — terdon, Commented Oct 20, 2022 at 16:54
If you think of regexp with regard to XML, just stop it right now. It's not gonna work strictly speaking. Read more here why: stackoverflow.com/a/1732454/2511795. — 0andriy, Commented Dec 7, 2023 at 15:49

jubilatious1 · Accepted Answer · 2022-10-21 04:10:56Z

Using Raku (formerly known as Perl_6)

~$ raku -MXML -e 'my $xml = open-xml($*ARGFILES.Str);  \
                  .say for $xml.getElementsByTagName("entry").grep(/ TSPAN6 | TNMD /).pairs;'  file.xml

#OR

~$ raku -MXML -e 'my @xml = open-xml($*ARGFILES.Str).getElementsByTagName("entry");  \
                  my @names = <TSPAN6 TNMD>; .say for @xml.grep(/@names/).pairs;' file.xml

It's far better in your case where you have an authentic XML file, to use an authentic XML parser. In the above code, you call the Raku Community Module XML with the commandline flag -MXML (BTW, which is identical to how you call modules at the commandline using Perl). Take a moment to familiarize yourself with the current XML schema, then plan your coding accordingly:

https://www.proteinatlas.org/download/proteinatlas.xsd

The first answer above opens your file into $xml, an XML Document. Then the XML Document is broken into (top-level) Elements named entry using .getElementsByTagName();. Finally, each element is grepped through to obtain elements containing the desired string(s).

The second answer above searches for XML Elements named entry and opens your file into @xml, a Raku array. Then each element is grepped through to obtain elements containing the desired string(s), strings which had been held in the @names array.

Sample Input (taken from @terdon's excellent answer):

<?xml version="1.0" encoding="UTF-8"?>
<proteinAtlas xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://v21.proteinatlas.org/download/proteinatlas.xsd" schemaVersion="2.6">
    <entry version="21.1" url="http://v21.proteinatlas.org/ENSG00000000003">
        <name>TSPAN6</name>
        <synonym>T245</synonym>
        <synonym>TM4SF6</synonym>
        <synonym>TSPAN-6</synonym>
        <identifier id="ENSG00000000003" db="Ensembl" version="103.38" assembly="GRCh38.p13" gencodeVersion="37">
            <xref id="O43657" db="Uniprot/SWISSPROT"/>
            <xref id="7105" db="NCBI GeneID"/>
        </identifier>
        <proteinClasses>
            <proteinClass source="MDM" id="Ma" parent_id="" name="Predicted membrane proteins"/>
            <proteinClass source="Ezkurdia et al 2014" id="Eb" parent_id="" name="Protein evidence (Ezkurdia et al 2014)"/>
        </proteinClasses>
        <proteinEvidence evidence="Evidence at protein level">
            <evidence source="HPA" evidence="Evidence at transcript level"/>
            <evidence source="MS" evidence="Not available"/>
            <evidence source="UniProt" evidence="Evidence at protein level"/>
        </proteinEvidence>
  </entry>
    <entry version="21.1" url="http://v21.proteinatlas.org/ENSG00000000005">
        <name>TNMD</name>
        <synonym>BRICD4</synonym>
        <synonym>ChM1L</synonym>
        <synonym>myodulin</synonym>
        <synonym>TEM</synonym>
        <synonym>tendin</synonym>
        <identifier id="ENSG00000000005" db="Ensembl" version="103.38" assembly="GRCh38.p13" gencodeVersion="37">
            <xref id="Q9H2S6" db="Uniprot/SWISSPROT"/>
            <xref id="64102" db="NCBI GeneID"/>
        </identifier>
        <proteinClasses>
            <proteinClass source="MDM" id="Ma" parent_id="" name="Predicted membrane proteins"/>
            <proteinClass source="MDM" id="Md" parent_id="" name="Membrane proteins predicted by MDM"/>
            <proteinClass source="MEMSAT3" id="Me" parent_id="" name="MEMSAT3 predicted membrane proteins"/>
        </proteinClasses>
        <proteinEvidence evidence="Evidence at protein level">
            <evidence source="HPA" evidence="Evidence at transcript level"/>
            <evidence source="MS" evidence="Not available"/>
            <evidence source="UniProt" evidence="Evidence at protein level"/>
        </proteinEvidence>
  </entry>
</proteinAtlas>

Sample Output:

0 => <entry version="21.1" url="http://v21.proteinatlas.org/ENSG00000000003">
        <name>TSPAN6</name>
        <synonym>T245</synonym>
        <synonym>TM4SF6</synonym>
        <synonym>TSPAN-6</synonym>
        <identifier version="103.38" gencodeVersion="37" assembly="GRCh38.p13" db="Ensembl" id="ENSG00000000003">
            <xref id="O43657" db="Uniprot/SWISSPROT"/>
            <xref db="NCBI GeneID" id="7105"/>
        </identifier>
        <proteinClasses>
            <proteinClass id="Ma" source="MDM" parent_id="" name="Predicted membrane proteins"/>
            <proteinClass name="Protein evidence (Ezkurdia et al 2014)" parent_id="" id="Eb" source="Ezkurdia et al 2014"/>
        </proteinClasses>
        <proteinEvidence evidence="Evidence at protein level">
            <evidence source="HPA" evidence="Evidence at transcript level"/>
            <evidence evidence="Not available" source="MS"/>
            <evidence evidence="Evidence at protein level" source="UniProt"/>
        </proteinEvidence>
  </entry>
1 => <entry version="21.1" url="http://v21.proteinatlas.org/ENSG00000000005">
        <name>TNMD</name>
        <synonym>BRICD4</synonym>
        <synonym>ChM1L</synonym>
        <synonym>myodulin</synonym>
        <synonym>TEM</synonym>
        <synonym>tendin</synonym>
        <identifier version="103.38" assembly="GRCh38.p13" db="Ensembl" gencodeVersion="37" id="ENSG00000000005">
            <xref id="Q9H2S6" db="Uniprot/SWISSPROT"/>
            <xref db="NCBI GeneID" id="64102"/>
        </identifier>
        <proteinClasses>
            <proteinClass name="Predicted membrane proteins" source="MDM" id="Ma" parent_id=""/>
            <proteinClass parent_id="" id="Md" name="Membrane proteins predicted by MDM" source="MDM"/>
            <proteinClass id="Me" name="MEMSAT3 predicted membrane proteins" parent_id="" source="MEMSAT3"/>
        </proteinClasses>
        <proteinEvidence evidence="Evidence at protein level">
            <evidence evidence="Evidence at transcript level" source="HPA"/>
            <evidence evidence="Not available" source="MS"/>
            <evidence evidence="Evidence at protein level" source="UniProt"/>
        </proteinEvidence>
  </entry>

Above, the call to pairs is simply to number the output elements. Finally, a comment by @terdon correctly states that grep for gene names may not be your safest bet. You can dramatically simplify your return values if you search for Ids instead (if indeed, the limited subset returned is useful to you):

~$ raku -MXML -e 'my $xml=open-xml($*ARGFILES.Str); put $xml.getElementById("ENSG00000000003"|"ENSG00000000005").pairs;' file.xml

Returns:

0   <identifier version="103.38" gencodeVersion="37" id="ENSG00000000003" db="Ensembl" assembly="GRCh38.p13">
            <xref id="O43657" db="Uniprot/SWISSPROT"/>
            <xref id="7105" db="NCBI GeneID"/>
        </identifier>
0   <identifier gencodeVersion="37" assembly="GRCh38.p13" version="103.38" db="Ensembl" id="ENSG00000000005">
            <xref db="Uniprot/SWISSPROT" id="Q9H2S6"/>
            <xref db="NCBI GeneID" id="64102"/>
        </identifier>

https://github.com/raku-community-modules/XML
https://rakudo.org/
https://raku.org

Kusalananda · Accepted Answer · 2022-10-21 11:49:30Z

Assuming the input is a well-formed XML document (like the example in terdon's answer, but not what's shown in the question), you may use xmlstarlet to output a copy of each entry node with a certain name and proteinClass's source attribute.

xmlstarlet select --template \
   --copy-of '//entry[name = "TNMD" and proteinClasses/proteinClass/@source = "MEMSAT3"]' \
   -nl file

This selects all entry nodes with a particular name that also has a proteinClasses/proteinClass sub-node with a particular source attribute value. A copy of each matching entry node will be outputted with a trailing newline added.

terdon · Accepted Answer · 2022-10-20 20:49:30Z

You can do this in GNU awk by using </entry[^>]*> as the record separator. For example, using this file as input:

<?xml version="1.0" encoding="UTF-8"?>
<proteinAtlas xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://v21.proteinatlas.org/download/proteinatlas.xsd" schemaVersion="2.6">
    <entry version="21.1" url="http://v21.proteinatlas.org/ENSG00000000003">
        <name>TSPAN6</name>
        <synonym>T245</synonym>
        <synonym>TM4SF6</synonym>
        <synonym>TSPAN-6</synonym>
        <identifier id="ENSG00000000003" db="Ensembl" version="103.38" assembly="GRCh38.p13" gencodeVersion="37">
            <xref id="O43657" db="Uniprot/SWISSPROT"/>
            <xref id="7105" db="NCBI GeneID"/>
        </identifier>
        <proteinClasses>
            <proteinClass source="MDM" id="Ma" parent_id="" name="Predicted membrane proteins"/>
            <proteinClass source="Ezkurdia et al 2014" id="Eb" parent_id="" name="Protein evidence (Ezkurdia et al 2014)"/>
        </proteinClasses>
        <proteinEvidence evidence="Evidence at protein level">
            <evidence source="HPA" evidence="Evidence at transcript level"/>
            <evidence source="MS" evidence="Not available"/>
            <evidence source="UniProt" evidence="Evidence at protein level"/>
        </proteinEvidence>
  </entry>
    <entry version="21.1" url="http://v21.proteinatlas.org/ENSG00000000005">
        <name>TNMD</name>
        <synonym>BRICD4</synonym>
        <synonym>ChM1L</synonym>
        <synonym>myodulin</synonym>
        <synonym>TEM</synonym>
        <synonym>tendin</synonym>
        <identifier id="ENSG00000000005" db="Ensembl" version="103.38" assembly="GRCh38.p13" gencodeVersion="37">
            <xref id="Q9H2S6" db="Uniprot/SWISSPROT"/>
            <xref id="64102" db="NCBI GeneID"/>
        </identifier>
        <proteinClasses>
            <proteinClass source="MDM" id="Ma" parent_id="" name="Predicted membrane proteins"/>
            <proteinClass source="MDM" id="Md" parent_id="" name="Membrane proteins predicted by MDM"/>
            <proteinClass source="MEMSAT3" id="Me" parent_id="" name="MEMSAT3 predicted membrane proteins"/>
        </proteinClasses>
        <proteinEvidence evidence="Evidence at protein level">
            <evidence source="HPA" evidence="Evidence at transcript level"/>
            <evidence source="MS" evidence="Not available"/>
            <evidence source="UniProt" evidence="Evidence at protein level"/>
        </proteinEvidence>
  </entry>
</proteinAtlas>

You could get the data for TNMD with:

$ gawk 'BEGIN{ RS="</entry[^>]*>" } /TNMD/' a

    <entry version="21.1" url="http://v21.proteinatlas.org/ENSG00000000005">
        <name>TNMD</name>
        <synonym>BRICD4</synonym>
        <synonym>ChM1L</synonym>
        <synonym>myodulin</synonym>
        <synonym>TEM</synonym>
        <synonym>tendin</synonym>
        <identifier id="ENSG00000000005" db="Ensembl" version="103.38" assembly="GRCh38.p13" gencodeVersion="37">
            <xref id="Q9H2S6" db="Uniprot/SWISSPROT"/>
            <xref id="64102" db="NCBI GeneID"/>
        </identifier>
        <proteinClasses>
            <proteinClass source="MDM" id="Ma" parent_id="" name="Predicted membrane proteins"/>
            <proteinClass source="MDM" id="Md" parent_id="" name="Membrane proteins predicted by MDM"/>
            <proteinClass source="MEMSAT3" id="Me" parent_id="" name="MEMSAT3 predicted membrane proteins"/>
        </proteinClasses>
        <proteinEvidence evidence="Evidence at protein level">
            <evidence source="HPA" evidence="Evidence at transcript level"/>
            <evidence source="MS" evidence="Not available"/>
            <evidence source="UniProt" evidence="Evidence at protein level"/>
        </proteinEvidence>

That simply means "print this line if it matches TNMD". Of course, it will also print if the line is something like 87% identity to TNMD and it's bound to break on all sorts of edge cases because we're not using a proper parser.

With a proper parser, you can specify exactly where the string should be.

This looks like its working really well, how would i give it more than one string. Could I pass more options, is this right: awk 'BEGIN{ RS="</entry" } /TNMD/ || /MEMSAT/' — Giles, Commented Oct 20, 2022 at 16:52
Yes. But you know, this is a really bad idea. Consider a simple example like MAPK and MAPKK. If you search for MAPK, you will also find MAPKK. And this is hardly a rare occurrence if you're working with HUGO names. If you can, try using the ENSEMBL identifiers or RefSeq or anything else that is less likely to be a substring of another identifier. Or, even better: use biopython or bioperl and a proper parser. You might want to ask over at Bioinformatics. — terdon, Commented Oct 20, 2022 at 16:57
It should never be accepted answer... Using regexp on XML data is so-o-o bad idea, see stackoverflow.com/a/1732454/2511795. (It's pity I can't vote down this) — 0andriy, Commented Dec 7, 2023 at 15:48

Stack Exchange Network

Identify strings between patterns and print entire region between pattern if string is found. Perferably using awk

3 Answers 3

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged
text-processing
awk
xml
bioinformatics
.

Hot Network Questions

Identify strings between patterns and print entire region between pattern if string is found. Perferably using awk

3 Answers 3

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged text-processingawkxmlbioinformatics.

Related

Hot Network Questions

Not the answer you're looking for? Browse other questions tagged
text-processing
awk
xml
bioinformatics
.