1

I'm working with several huge (>2gb) XML files and their size is causing problems.

(My application uses XMLReader in a PHP script to parse smaller ~500mb files, and that works fine, but XMLReader won't open these large files.)

So - my idea is to eliminate big parent nodes of the file that I know I don't need.

For example, if the structure of the file looks like this:

<record id="1">
    <a>
        <detail>blah</detail>
        ....
        <detail>blah</detail>
    </a>
    <b>
        <detail>blah</detail>
        ....
        <detail>blah</detail>
    </b>
    <c>
        <detail>blah</detail>
        ....
        <detail>blah</detail>
    </c>
</record>
...
<record id="999999">
    <a>
        <detail>blah</detail>
        ....
        <detail>blah</detail>
    </a>
    <b>
        <detail>blah</detail>
        ....
        <detail>blah</detail>
    </b>
    <c>
        <detail>blah</detail>
        ....
        <detail>blah</detail>
    </c>
</record>

For my purposes - I only need the data in parent node <a> for each record. If I could eliminate parent nodes <b> and <c> from every record, I could reduce the size of the file substantially, so it would be small enough to work with normally.

What's the best way to do something like this?

Most of the "XML-aware" utilities I've tried choke on a file this large, so I'm hoping that I can do this with something like sed or grep.

6
  • 1
    XML, like (X)HTML is not a regular language; as a result parsing or manipulating it with regex is not feasible. You may be able to do some minimal manipulation with awk or sed, but you should not rely on regex to allow the manipulation or parsing of arbitrary XML.
    – HalosGhost
    Sep 18 '14 at 17:28
  • If you like perl: search.cpan.org/~mirod/XML-Twig-3.48/Twig.pm was designed to handle huge XML files Sep 18 '14 at 17:55
  • @HalosGhost - Thanks for your comment. Just curious though... Why can't you manipulate it with regex? For example, wouldn't <b>(.|\n)*?</b> match all of the <b> nodes, regardless of what else was within them? Sep 18 '14 at 18:10
  • Sure, but the trouble becomes really obvious once you see attributes (<b test-attrib="things">) or self closing tags (<hr />). Regex are only truly functional on regular languages; XML/(X)HTML are not regular languages.
    – HalosGhost
    Sep 18 '14 at 18:25
  • have you a programm call xsltproc ? if yes, xml/xsl master somewhere (or me tomorrow) can produce a parser to fit your need.
    – Archemar
    Sep 18 '14 at 21:18
1

You may use awk:

$cat my.xml | awk '/<b>/{hide=1} /<\/record>/ {hide=0} {if (hide==0) print;}' >mynew.xml

this will hide everything since line which contains <b> and start display with line containing </record>

Per your comment, if your XML is one big line - just split it to lines and remove newline chars back after you've done the conversion.

$cat my.xml|sed 's/>/>\n/g'| awk ....... | tr -d '\n' >.....

Throw away xml, start using YAML or JSON!

2
  • this is an elegant solution. However, made a mistake in my original post... My XML doesn't actually have line returns after each node (I just showed it that way for readability). My actual XML file flows continuously on one line. (However, oddly, there are a handful of randomly scattered line breaks throughout.) So, I need a solution that doesn't depend on line-breaks. Sep 19 '14 at 15:40
  • Just add line-break after each > and then remove all breaks (updated answer).
    – gena2x
    Sep 19 '14 at 16:33
1

Personnaly, I would do something in C (which is at lowest level possible before assembly) and Loop through all the nodes using libxml.

Here are some examples : http://www.xmlsoft.org/examples/

Use GCC to compile your code.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.