3

I have a text file, i want to search for tags such as the following:

<category="SpecificDisease">Type II human complement C2 deficiency</category>
<category="Modifier">Huntington disease</category>
<category="CompositeMention">hereditary breast and ovarian cancer</category>
<category="DiseaseClass">myopathy</category>

and produce the following and write them to a new text file.

Type II human complement C2 deficiency
Huntington disease
hereditary breast and ovarian cancer
myopathy
  • 3
    Does your text file contain other tags? What is the first line if the text file? Is your text file actually an XML document? – AndrewNimmo Nov 6 '13 at 19:05
  • 3
    Unless you add more details as requested by @AndrewNimmo, even awk -F'<|>' '$0=$3' is enough. – manatwork Nov 6 '13 at 19:19
  • Hopefully the point is clear here that if the file has more complicated stuff in it than just the lines you are looking for, that will make a significant difference because the task is more complicated. If it doesn't, it's fairly simple. – goldilocks Nov 6 '13 at 19:25
  • @manatwork You should post your comment as an answer. It is a clear winner, golf-wise :) – Joseph R. Nov 6 '13 at 19:38
  • My file is a text file, the following is part of the file: – nlp Nov 6 '13 at 20:56
7

That looks like a kind of XML or similar markup language file. Such files should not be parsed by simple regular expressions for fear of waking TO͇̹̺ͅƝ̴ȳ̳ TH̘Ë͖́̉ ͠P̯͍̭O̚​N̐Y̡. You should use a parser specific for that markup and your favorite scripting language instead.

That looks like OMIM or HPO data in which case you should be able to get simple text files and simplify things. If you can't and really really need to parse this file, you could do this in perl:

perl -lne '/<.*?>([^<>]+)/ && print $1' foo.txt

However, this will break if you have multiple tags per line, or if your tag's contents can span multiple lines or if the tag's data can contain > or <. If all your info is always between <category="whatever">blah blah</category>, you can get everything more robustly (including multi line tag contents and embedded < or >):

#!/usr/bin/env perl

## Set the start and end tags
$end="</category>"; 
$start="<category=.*?>"; 

## Read through the file line by line
while(<>){
    ## set $a to one if the current line matches $start
    $a=1 if /$start/; 
    ## If the current line matches $start, capture any relevant content.
    ## I am also removing any $start or $end tags if present.
    if(s/($start)*(.+)($end)*/$2/){
    push @lines,$2 if $a==1;
    }  
    ## If the current line matches $end, capture any relevant content,
    ## print what we have saved so far, set $a back to 0 and empty the
    ## @lines array
    if(/$end/){
    map{s/$end//;}@lines; 
    print "@lines\n";
    @lines=(); 
    $a=0
    }; 
}

Save this script as foo.pl or whatever, make it executable and run it on your file:

./foo.pl file.txt

For example:

$ cat file.txt 
<category="SpecificDisease">Type II 
 human complement C2 deficiency</category>
<category="Modifier">Huntington disease</category>
<category="CompositeMention">hereditary breast < and ovarian cancer</category>
<category="DiseaseClass">myopathy > cardiopathy</category>

$ ./foo.pl file.txt 
Type II   human complement C2 deficiency
Huntington disease
hereditary breast < and ovarian cancer
myopathy > cardiopathy

Once more though, I stress that if (which is very likely) your file is more complex than the example above, this will fail and more sophisticated methods will be needed.

  • +1 for referencing that (in)famous SO answer. Just to clarify for the OP: the reason the for the infamous Q&A is the tendency of people to insist regexps are fine because they are too lazy to learn how to use a parser, and the point is that you can do it properly (with a parser), or you can take your chances (with a regexp). None-the-less, as terdon says, if you are sure the files you are parsing always have the same identical structure, a regexp is viable. Just don't use them generally with complete HTML/XML documents. – goldilocks Nov 6 '13 at 19:23
  • @goldilocks Not to contradict your (very valid) point, but I believe that Perl's (ir)regular expressions with their many bells and whistles can be used to create a grammar that can effectively parse (well-formed) markup files. Please correct me if I'm wrong and you have experience to the contrary. – Joseph R. Nov 6 '13 at 19:36
  • 1
    Not sure which XML parser could be used. The code posted in the question is clearly not XML. – manatwork Nov 6 '13 at 19:37
  • @JosephR. I think it depends on what you are trying to extract -- there are inevitably specific cases that can be done with a regexp, then you have a counter-case that can't -- so you refine the regexp (nest them! deeply!) -- then there's another variant counter-case, etc. Since perl also has HTML::Parser, I don't see much point in the debate. Do I need to prove I can climb a wall blindfolded with one arm behind my back when there's a ladder available and no need for the handicaps? If I could, would I then recommend it in practice vs. the ladder? Nah. – goldilocks Nov 6 '13 at 19:53
  • @manatwork Good catch. – goldilocks Nov 6 '13 at 19:56
4

You can do this simply in sed:

sed 's/>\([^>]\+\)</\1/' your_file > new_file

to extract the text between > and < on each line. This assumes you have one tag per line.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.